public inbox for [email protected]help / color / mirror / Atom feed
[PATCH 3/5] Extended statistics on expressions 6+ messages / 4 participants [nested] [flat]
* [PATCH 3/5] Extended statistics on expressions @ 2020-12-03 15:19 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 6+ messages in thread From: Tomas Vondra @ 2020-12-03 15:19 UTC (permalink / raw) Allow defining extended statistics on expressions, not just simple column references. With this commit, it's possible to do things like CREATE TABLE t (a int); CREATE STATISTICS s ON mod(a,10), mod(a,20) FROM t; and the collected statistics will be useful for estimating queries using those expressions in various places, like SELECT * FROM t WHERE mod(a,10) = 0 AND mod(a,20) = 0; or SELECT mod(a,10), mod(a,20) FROM t GROUP BY 1, 2; The commit also adds a new statistics type "expressions" which builds the usual per-column statistics for each expression, allowing better estimates even for queries with just a single expression, which are not affected by multi-column statistics. This achieves the same goal as creating expression indexes, without index maintenance overhead. --- doc/src/sgml/catalogs.sgml | 238 +++- doc/src/sgml/ref/create_statistics.sgml | 98 +- src/backend/catalog/Makefile | 8 +- src/backend/catalog/system_views.sql | 69 + src/backend/commands/statscmds.c | 319 +++-- src/backend/nodes/copyfuncs.c | 14 + src/backend/nodes/equalfuncs.c | 13 + src/backend/nodes/outfuncs.c | 12 + src/backend/optimizer/util/plancat.c | 62 + src/backend/parser/gram.y | 38 +- src/backend/parser/parse_agg.c | 10 + src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + src/backend/parser/parse_utilcmd.c | 121 +- src/backend/statistics/dependencies.c | 369 ++++- src/backend/statistics/extended_stats.c | 1202 +++++++++++++++-- src/backend/statistics/mcv.c | 396 +++--- src/backend/statistics/mvdistinct.c | 101 +- src/backend/tcop/utility.c | 23 +- src/backend/utils/adt/ruleutils.c | 269 +++- src/backend/utils/adt/selfuncs.c | 447 +++++- src/bin/pg_dump/t/002_pg_dump.pl | 12 + src/bin/psql/describe.c | 66 +- src/include/catalog/pg_proc.dat | 8 + src/include/catalog/pg_statistic_ext.h | 4 + src/include/catalog/pg_statistic_ext_data.h | 1 + src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 16 + src/include/nodes/pathnodes.h | 3 +- src/include/parser/parse_node.h | 1 + src/include/parser/parse_utilcmd.h | 2 + .../statistics/extended_stats_internal.h | 38 +- src/include/statistics/statistics.h | 2 + .../regress/expected/create_table_like.out | 20 +- src/test/regress/expected/oidjoins.out | 10 +- src/test/regress/expected/rules.out | 73 + src/test/regress/expected/stats_ext.out | 681 +++++++++- src/test/regress/sql/create_table_like.sql | 2 + src/test/regress/sql/stats_ext.sql | 313 ++++- 39 files changed, 4477 insertions(+), 594 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..dfe0b8917b 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7358,7 +7358,7 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <structfield>stxkind</structfield> <type>char[]</type> </para> <para> - An array containing codes for the enabled statistic kinds; + An array containing codes for the enabled statistics kinds; valid values are: <literal>d</literal> for n-distinct statistics, <literal>f</literal> for functional dependency statistics, and @@ -9412,6 +9412,11 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <entry>extended planner statistics</entry> </row> + <row> + <entry><link linkend="view-pg-stats-ext-exprs"><structname>pg_stats_ext_exprs</structname></link></entry> + <entry>extended planner statistics for expressions</entry> + </row> + <row> <entry><link linkend="view-pg-tables"><structname>pg_tables</structname></link></entry> <entry>tables</entry> @@ -12997,6 +13002,237 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx </sect1> + <sect1 id="view-pg-stats-ext-exprs"> + <title><structname>pg_stats_ext_exprs</structname></title> + + <indexterm zone="view-pg-stats-ext-exprs"> + <primary>pg_stats_ext_exprs</primary> + </indexterm> + + <para> + The view <structname>pg_stats_ext_exprs</structname> provides access to + the information stored in the <link + linkend="catalog-pg-statistic-ext"><structname>pg_statistic_ext</structname></link> + and <link linkend="catalog-pg-statistic-ext-data"><structname>pg_statistic_ext_data</structname></link> + catalogs. This view allows access only to rows of + <link linkend="catalog-pg-statistic-ext"><structname>pg_statistic_ext</structname></link> and <link linkend="catalog-pg-statistic-ext-data"><structname>pg_statistic_ext_data</structname></link> + that correspond to tables the user has permission to read, and therefore + it is safe to allow public read access to this view. + </para> + + <para> + <structname>pg_stats_ext_exprs</structname> is also designed to present + the information in a more readable format than the underlying catalogs + — at the cost that its schema must be extended whenever the structure + of statistics in <link linkend="catalog-pg-statistic"><structname>pg_statistic</structname></link> changes. + </para> + + <table> + <title><structname>pg_stats_ext_exprs</structname> Columns</title> + <tgroup cols="1"> + <thead> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + Column Type + </para> + <para> + Description + </para></entry> + </row> + </thead> + + <tbody> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>schemaname</structfield> <type>name</type> + (references <link linkend="catalog-pg-namespace"><structname>pg_namespace</structname></link>.<structfield>nspname</structfield>) + </para> + <para> + Name of schema containing table + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>tablename</structfield> <type>name</type> + (references <link linkend="catalog-pg-class"><structname>pg_class</structname></link>.<structfield>relname</structfield>) + </para> + <para> + Name of table + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>statistics_schemaname</structfield> <type>name</type> + (references <link linkend="catalog-pg-namespace"><structname>pg_namespace</structname></link>.<structfield>nspname</structfield>) + </para> + <para> + Name of schema containing extended statistic + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>statistics_name</structfield> <type>name</type> + (references <link linkend="catalog-pg-statistic-ext"><structname>pg_statistic_ext</structname></link>.<structfield>stxname</structfield>) + </para> + <para> + Name of extended statistics + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>statistics_owner</structfield> <type>name</type> + (references <link linkend="catalog-pg-authid"><structname>pg_authid</structname></link>.<structfield>rolname</structfield>) + </para> + <para> + Owner of the extended statistics + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>expr</structfield> <type>text</type> + </para> + <para> + Expression the extended statistics is defined on + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>null_frac</structfield> <type>float4</type> + </para> + <para> + Fraction of column entries that are null + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>avg_width</structfield> <type>int4</type> + </para> + <para> + Average width in bytes of column's entries + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>n_distinct</structfield> <type>float4</type> + </para> + <para> + If greater than zero, the estimated number of distinct values in the + column. If less than zero, the negative of the number of distinct + values divided by the number of rows. (The negated form is used when + <command>ANALYZE</command> believes that the number of distinct values is + likely to increase as the table grows; the positive form is used when + the column seems to have a fixed number of possible values.) For + example, -1 indicates a unique column in which the number of distinct + values is the same as the number of rows. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>most_common_vals</structfield> <type>anyarray</type> + </para> + <para> + A list of the most common values in the column. (Null if + no values seem to be more common than any others.) + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>most_common_freqs</structfield> <type>float4[]</type> + </para> + <para> + A list of the frequencies of the most common values, + i.e., number of occurrences of each divided by total number of rows. + (Null when <structfield>most_common_vals</structfield> is.) + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>histogram_bounds</structfield> <type>anyarray</type> + </para> + <para> + A list of values that divide the column's values into groups of + approximately equal population. The values in + <structfield>most_common_vals</structfield>, if present, are omitted from this + histogram calculation. (This column is null if the column data type + does not have a <literal><</literal> operator or if the + <structfield>most_common_vals</structfield> list accounts for the entire + population.) + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>correlation</structfield> <type>float4</type> + </para> + <para> + Statistical correlation between physical row ordering and + logical ordering of the column values. This ranges from -1 to +1. + When the value is near -1 or +1, an index scan on the column will + be estimated to be cheaper than when it is near zero, due to reduction + of random access to the disk. (This column is null if the column data + type does not have a <literal><</literal> operator.) + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>most_common_elems</structfield> <type>anyarray</type> + </para> + <para> + A list of non-null element values most often appearing within values of + the column. (Null for scalar types.) + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>most_common_elem_freqs</structfield> <type>float4[]</type> + </para> + <para> + A list of the frequencies of the most common element values, i.e., the + fraction of rows containing at least one instance of the given value. + Two or three additional values follow the per-element frequencies; + these are the minimum and maximum of the preceding per-element + frequencies, and optionally the frequency of null elements. + (Null when <structfield>most_common_elems</structfield> is.) + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>elem_count_histogram</structfield> <type>float4[]</type> + </para> + <para> + A histogram of the counts of distinct non-null element values within the + values of the column, followed by the average number of distinct + non-null elements. (Null for scalar types.) + </para></entry> + </row> + </tbody> + </tgroup> + </table> + + <para> + The maximum number of entries in the array fields can be controlled on a + column-by-column basis using the <link linkend="sql-altertable"><command>ALTER + TABLE SET STATISTICS</command></link> + command, or globally by setting the + <xref linkend="guc-default-statistics-target"/> run-time parameter. + </para> + + </sect1> + <sect1 id="view-pg-tables"> <title><structname>pg_tables</structname></title> diff --git a/doc/src/sgml/ref/create_statistics.sgml b/doc/src/sgml/ref/create_statistics.sgml index 4363be50c3..ba50ee6bcd 100644 --- a/doc/src/sgml/ref/create_statistics.sgml +++ b/doc/src/sgml/ref/create_statistics.sgml @@ -21,9 +21,13 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> +CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="parameter">statistics_name</replaceable> + ON ( <replaceable class="parameter">expression</replaceable> ) + FROM <replaceable class="parameter">table_name</replaceable> + CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="parameter">statistics_name</replaceable> [ ( <replaceable class="parameter">statistics_kind</replaceable> [, ... ] ) ] - ON <replaceable class="parameter">column_name</replaceable>, <replaceable class="parameter">column_name</replaceable> [, ...] + ON { <replaceable class="parameter">column_name</replaceable> | ( <replaceable class="parameter">expression</replaceable> ) } [, ...] FROM <replaceable class="parameter">table_name</replaceable> </synopsis> @@ -39,6 +43,16 @@ CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="parameter">statistics_na database and will be owned by the user issuing the command. </para> + <para> + The <command>CREATE STATISTICS</command> command has two basic forms. The + simple variant allows building statistics for a single expression, does + not allow specifying any statistics kinds and provides benefits similar + to an expression index. The full variant allows defining statistics objects + on multiple columns and expressions, and selecting which statistics kinds will + be built. The per-expression statistics are built automatically when there + is at least one expression. + </para> + <para> If a schema name is given (for example, <literal>CREATE STATISTICS myschema.mystat ...</literal>) then the statistics object is created in the @@ -86,7 +100,9 @@ CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="parameter">statistics_na dependency statistics, and <literal>mcv</literal> which enables most-common values lists. If this clause is omitted, all supported statistics kinds are - included in the statistics object. + included in the statistics object. Expression statistics are built + automatically when the statistics definition includes complex + expressions and not just simple column references. For more information, see <xref linkend="planner-stats-extended"/> and <xref linkend="multivariate-statistics-examples"/>. </para> @@ -104,6 +120,17 @@ CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="parameter">statistics_na </listitem> </varlistentry> + <varlistentry> + <term><replaceable class="parameter">expression</replaceable></term> + <listitem> + <para> + The expression to be covered by the computed statistics. In this case + only a single expression is required, in which case only statistics + for the expression are built. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><replaceable class="parameter">table_name</replaceable></term> <listitem> @@ -125,6 +152,13 @@ CREATE STATISTICS [ IF NOT EXISTS ] <replaceable class="parameter">statistics_na reading it. Once created, however, the ownership of the statistics object is independent of the underlying table(s). </para> + + <para> + Expression statistics are per-expression and are similar to creating an + index on the expression, except that they avoid the overhead of index + maintenance. Expression statistics are built automatically when there + is at least one expression in the statistics object definition. + </para> </refsect1> <refsect1 id="sql-createstatistics-examples"> @@ -196,6 +230,66 @@ EXPLAIN ANALYZE SELECT * FROM t2 WHERE (a = 1) AND (b = 2); in the table, allowing it to generate better estimates in both cases. </para> + <para> + Create table <structname>t3</structname> with a single timestamp column, + and run a query using an expression on that column. Without extended + statistics, the planner has no information about data distribution for + results of those expression, and uses default estimates as illustrated + by the first query. The planner also does not realize that the value of + the second column fully defines the value of the other column, because + date truncated to day still identifies the month. Then expression and + ndistinct statistics are built on those two columns: + +<programlisting> +CREATE TABLE t3 ( + a timestamp +); + +INSERT INTO t3 SELECT i FROM generate_series('2020-01-01'::timestamp, + '2020-12-31'::timestamp, + '1 minute'::interval) s(i); + +ANALYZE t3; + +-- the number of matching rows will be drastically underestimated: +EXPLAIN ANALYZE SELECT * FROM t3 + WHERE date_trunc('month', a) = '2020-01-01'::timestamp; + +EXPLAIN ANALYZE SELECT * FROM t3 + WHERE date_trunc('day', a) BETWEEN '2020-01-01'::timestamp + AND '2020-06-30'::timestamp; + +EXPLAIN ANALYZE SELECT date_trunc('month', a), date_trunc('day', a) + FROM t3 GROUP BY 1, 2; + +-- per-expression statistics are built automatically +CREATE STATISTICS s3 (ndistinct) ON date_trunc('month', a), date_trunc('day', a) FROM t3; + +ANALYZE t1; + +-- now the row count estimates are more accurate: +EXPLAIN ANALYZE SELECT * FROM t3 + WHERE date_trunc('month', a) = '2020-01-01'::timestamp; + +EXPLAIN ANALYZE SELECT * FROM t3 + WHERE date_trunc('day', a) BETWEEN '2020-01-01'::timestamp + AND '2020-06-30'::timestamp; + +EXPLAIN ANALYZE SELECT date_trunc('month', a), date_trunc('day', a) + FROM t3 GROUP BY 1, 2; +</programlisting> + + Without expression and ndistinct statistics, the planner would assume + that the two <literal>WHERE</literal> and <literal>GROUP BY</literal> + conditions are independent, and would multiply their selectivities + together to arrive at a much-too-small row count estimate in the first + two queries, and a much-too-high group count estimate in the aggregate + query. This is further exacerbated by the lack of accurate statistics + for the expressions, forcing the planner to use default selectivities. + With such statistics, the planner recognizes that the conditions are + correlated and arrives at much more accurate estimates. + </para> + </refsect1> <refsect1> diff --git a/src/backend/catalog/Makefile b/src/backend/catalog/Makefile index 70bc2123df..e36a9602c1 100644 --- a/src/backend/catalog/Makefile +++ b/src/backend/catalog/Makefile @@ -49,15 +49,15 @@ include $(top_srcdir)/src/backend/common.mk # Note: the order of this list determines the order in which the catalog # header files are assembled into postgres.bki. BKI_BOOTSTRAP catalogs -# must appear first, and there are reputedly other, undocumented ordering -# dependencies. +# must appear first, and pg_statistic before pg_statistic_ext_data, and +# there are reputedly other, undocumented ordering dependencies. CATALOG_HEADERS := \ pg_proc.h pg_type.h pg_attribute.h pg_class.h \ pg_attrdef.h pg_constraint.h pg_inherits.h pg_index.h pg_operator.h \ pg_opfamily.h pg_opclass.h pg_am.h pg_amop.h pg_amproc.h \ pg_language.h pg_largeobject_metadata.h pg_largeobject.h pg_aggregate.h \ - pg_statistic_ext.h pg_statistic_ext_data.h \ - pg_statistic.h pg_rewrite.h pg_trigger.h pg_event_trigger.h pg_description.h \ + pg_statistic.h pg_statistic_ext.h pg_statistic_ext_data.h \ + pg_rewrite.h pg_trigger.h pg_event_trigger.h pg_description.h \ pg_cast.h pg_enum.h pg_namespace.h pg_conversion.h pg_depend.h \ pg_database.h pg_db_role_setting.h pg_tablespace.h \ pg_authid.h pg_auth_members.h pg_shdepend.h pg_shdescription.h \ diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index fc94a73a54..281127b15c 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -264,6 +264,7 @@ CREATE VIEW pg_stats_ext WITH (security_barrier) AS JOIN pg_attribute a ON (a.attrelid = s.stxrelid AND a.attnum = k) ) AS attnames, + pg_get_statisticsobjdef_expressions(s.oid) as exprs, s.stxkind AS kinds, sd.stxdndistinct AS n_distinct, sd.stxddependencies AS dependencies, @@ -290,6 +291,74 @@ CREATE VIEW pg_stats_ext WITH (security_barrier) AS WHERE NOT has_column_privilege(c.oid, a.attnum, 'select') ) AND (c.relrowsecurity = false OR NOT row_security_active(c.oid)); +CREATE VIEW pg_stats_ext_exprs WITH (security_barrier) AS + SELECT cn.nspname AS schemaname, + c.relname AS tablename, + sn.nspname AS statistics_schemaname, + s.stxname AS statistics_name, + pg_get_userbyid(s.stxowner) AS statistics_owner, + stat.expr, + (stat.a).stanullfrac AS null_frac, + (stat.a).stawidth AS avg_width, + (stat.a).stadistinct AS n_distinct, + (CASE + WHEN (stat.a).stakind1 = 1 THEN (stat.a).stavalues1 + WHEN (stat.a).stakind2 = 1 THEN (stat.a).stavalues2 + WHEN (stat.a).stakind3 = 1 THEN (stat.a).stavalues3 + WHEN (stat.a).stakind4 = 1 THEN (stat.a).stavalues4 + WHEN (stat.a).stakind5 = 1 THEN (stat.a).stavalues5 + END) AS most_common_vals, + (CASE + WHEN (stat.a).stakind1 = 1 THEN (stat.a).stanumbers1 + WHEN (stat.a).stakind2 = 1 THEN (stat.a).stanumbers2 + WHEN (stat.a).stakind3 = 1 THEN (stat.a).stanumbers3 + WHEN (stat.a).stakind4 = 1 THEN (stat.a).stanumbers4 + WHEN (stat.a).stakind5 = 1 THEN (stat.a).stanumbers5 + END) AS most_common_freqs, + (CASE + WHEN (stat.a).stakind1 = 2 THEN (stat.a).stavalues1 + WHEN (stat.a).stakind2 = 2 THEN (stat.a).stavalues2 + WHEN (stat.a).stakind3 = 2 THEN (stat.a).stavalues3 + WHEN (stat.a).stakind4 = 2 THEN (stat.a).stavalues4 + WHEN (stat.a).stakind5 = 2 THEN (stat.a).stavalues5 + END) AS histogram_bounds, + (CASE + WHEN (stat.a).stakind1 = 3 THEN (stat.a).stanumbers1[1] + WHEN (stat.a).stakind2 = 3 THEN (stat.a).stanumbers2[1] + WHEN (stat.a).stakind3 = 3 THEN (stat.a).stanumbers3[1] + WHEN (stat.a).stakind4 = 3 THEN (stat.a).stanumbers4[1] + WHEN (stat.a).stakind5 = 3 THEN (stat.a).stanumbers5[1] + END) correlation, + (CASE + WHEN (stat.a).stakind1 = 4 THEN (stat.a).stavalues1 + WHEN (stat.a).stakind2 = 4 THEN (stat.a).stavalues2 + WHEN (stat.a).stakind3 = 4 THEN (stat.a).stavalues3 + WHEN (stat.a).stakind4 = 4 THEN (stat.a).stavalues4 + WHEN (stat.a).stakind5 = 4 THEN (stat.a).stavalues5 + END) AS most_common_elems, + (CASE + WHEN (stat.a).stakind1 = 4 THEN (stat.a).stanumbers1 + WHEN (stat.a).stakind2 = 4 THEN (stat.a).stanumbers2 + WHEN (stat.a).stakind3 = 4 THEN (stat.a).stanumbers3 + WHEN (stat.a).stakind4 = 4 THEN (stat.a).stanumbers4 + WHEN (stat.a).stakind5 = 4 THEN (stat.a).stanumbers5 + END) AS most_common_elem_freqs, + (CASE + WHEN (stat.a).stakind1 = 5 THEN (stat.a).stanumbers1 + WHEN (stat.a).stakind2 = 5 THEN (stat.a).stanumbers2 + WHEN (stat.a).stakind3 = 5 THEN (stat.a).stanumbers3 + WHEN (stat.a).stakind4 = 5 THEN (stat.a).stanumbers4 + WHEN (stat.a).stakind5 = 5 THEN (stat.a).stanumbers5 + END) AS elem_count_histogram + FROM pg_statistic_ext s JOIN pg_class c ON (c.oid = s.stxrelid) + LEFT JOIN pg_statistic_ext_data sd ON (s.oid = sd.stxoid) + LEFT JOIN pg_namespace cn ON (cn.oid = c.relnamespace) + LEFT JOIN pg_namespace sn ON (sn.oid = s.stxnamespace) + JOIN LATERAL ( + SELECT unnest(pg_get_statisticsobjdef_expressions(s.oid)) AS expr, + unnest(sd.stxdexpr)::pg_statistic AS a + ) stat ON (stat.expr IS NOT NULL); + -- unprivileged users may read pg_statistic_ext but not pg_statistic_ext_data REVOKE ALL on pg_statistic_ext_data FROM public; diff --git a/src/backend/commands/statscmds.c b/src/backend/commands/statscmds.c index 2bae205845..7370af820f 100644 --- a/src/backend/commands/statscmds.c +++ b/src/backend/commands/statscmds.c @@ -29,6 +29,8 @@ #include "commands/comment.h" #include "commands/defrem.h" #include "miscadmin.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/optimizer.h" #include "statistics/statistics.h" #include "utils/builtins.h" #include "utils/fmgroids.h" @@ -62,7 +64,8 @@ ObjectAddress CreateStatistics(CreateStatsStmt *stmt) { int16 attnums[STATS_MAX_DIMENSIONS]; - int numcols = 0; + int nattnums = 0; + int numcols; char *namestr; NameData stxname; Oid statoid; @@ -74,21 +77,25 @@ CreateStatistics(CreateStatsStmt *stmt) Datum datavalues[Natts_pg_statistic_ext_data]; bool datanulls[Natts_pg_statistic_ext_data]; int2vector *stxkeys; + List *stxexprs = NIL; + Datum exprsDatum; Relation statrel; Relation datarel; Relation rel = NULL; Oid relid; ObjectAddress parentobject, myself; - Datum types[3]; /* one for each possible type of statistic */ + Datum types[4]; /* one for each possible type of statistic */ int ntypes; ArrayType *stxkind; bool build_ndistinct; bool build_dependencies; bool build_mcv; + bool build_expressions; bool requested_type = false; int i; ListCell *cell; + ListCell *cell2; Assert(IsA(stmt, CreateStatsStmt)); @@ -190,72 +197,169 @@ CreateStatistics(CreateStatsStmt *stmt) } /* - * Currently, we only allow simple column references in the expression - * list. That will change someday, and again the grammar already supports - * it so we have to enforce restrictions here. For now, we can convert - * the expression list to a simple array of attnums. While at it, enforce - * some constraints. + * Make sure no more than STATS_MAX_DIMENSIONS columns are used. There + * might be duplicates and so on, but we'll deal with those later. + */ + numcols = list_length(stmt->exprs); + if (numcols > STATS_MAX_DIMENSIONS) + ereport(ERROR, + (errcode(ERRCODE_TOO_MANY_COLUMNS), + errmsg("cannot have more than %d columns in statistics", + STATS_MAX_DIMENSIONS))); + + /* + * Convert the expression list to a simple array of attnums, but also + * keep a list of more complex expressions. While at it, enforce some + * constraints. */ foreach(cell, stmt->exprs) { Node *expr = (Node *) lfirst(cell); - ColumnRef *cref; - char *attname; + StatsElem *selem; HeapTuple atttuple; Form_pg_attribute attForm; TypeCacheEntry *type; - if (!IsA(expr, ColumnRef)) + /* + * XXX How could we get anything else than a StatsElem, given the + * grammar? But let's keep it as a safety, maybe shall we turn it + * into an assert? + */ + if (!IsA(expr, StatsElem)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("only simple column references are allowed in CREATE STATISTICS"))); - cref = (ColumnRef *) expr; + errmsg("only simple column references and expressions are allowed in CREATE STATISTICS"))); - if (list_length(cref->fields) != 1) - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("only simple column references are allowed in CREATE STATISTICS"))); - attname = strVal((Value *) linitial(cref->fields)); + selem = (StatsElem *) expr; - atttuple = SearchSysCacheAttName(relid, attname); - if (!HeapTupleIsValid(atttuple)) - ereport(ERROR, - (errcode(ERRCODE_UNDEFINED_COLUMN), - errmsg("column \"%s\" does not exist", - attname))); - attForm = (Form_pg_attribute) GETSTRUCT(atttuple); + if (selem->name) /* column reference */ + { + char *attname; + attname = selem->name; + + atttuple = SearchSysCacheAttName(relid, attname); + if (!HeapTupleIsValid(atttuple)) + ereport(ERROR, + (errcode(ERRCODE_UNDEFINED_COLUMN), + errmsg("column \"%s\" does not exist", + attname))); + attForm = (Form_pg_attribute) GETSTRUCT(atttuple); + + /* Disallow use of system attributes in extended stats */ + if (attForm->attnum <= 0) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("statistics creation on system columns is not supported"))); + + /* Disallow data types without a less-than operator */ + type = lookup_type_cache(attForm->atttypid, TYPECACHE_LT_OPR); + if (type->lt_opr == InvalidOid) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column \"%s\" cannot be used in statistics because its type %s has no default btree operator class", + attname, format_type_be(attForm->atttypid)))); + + attnums[nattnums] = attForm->attnum; + nattnums++; + ReleaseSysCache(atttuple); + } + else /* expression */ + { + Node *expr = selem->expr; + Oid atttype; + + Assert(expr != NULL); + + /* + * Disallow data types without a less-than operator. + * + * We ignore this for statistics on a single expression, in + * which case we'll build the regular statistics only (and + * that code can deal with such data types). + */ + if (list_length(stmt->exprs) > 1) + { + atttype = exprType(expr); + type = lookup_type_cache(atttype, TYPECACHE_LT_OPR); + if (type->lt_opr == InvalidOid) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("expression cannot be used in statistics because its type %s has no default btree operator class", + format_type_be(atttype)))); + } + + stxexprs = lappend(stxexprs, expr); + } + } - /* Disallow use of system attributes in extended stats */ - if (attForm->attnum <= 0) + /* + * Parse the statistics kinds. Firstly, check that this is not the + * variant building statistics for a single expression, in which case + * we don't allow specifying any statistics kinds. The simple variant + * only has one expression, and does not allow statistics kinds. + */ + if ((list_length(stmt->exprs) == 1) && (list_length(stxexprs) == 1)) + { + /* statistics kinds not specified */ + if (list_length(stmt->stat_types) > 0) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("statistics creation on system columns is not supported"))); + errmsg("when building statistics on a single expression, statistics kinds may not be specified"))); + } - /* Disallow data types without a less-than operator */ - type = lookup_type_cache(attForm->atttypid, TYPECACHE_LT_OPR); - if (type->lt_opr == InvalidOid) - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("column \"%s\" cannot be used in statistics because its type %s has no default btree operator class", - attname, format_type_be(attForm->atttypid)))); + /* OK, let's check that we recognize the statistics kinds. */ + build_ndistinct = false; + build_dependencies = false; + build_mcv = false; + foreach(cell, stmt->stat_types) + { + char *type = strVal((Value *) lfirst(cell)); - /* Make sure no more than STATS_MAX_DIMENSIONS columns are used */ - if (numcols >= STATS_MAX_DIMENSIONS) + if (strcmp(type, "ndistinct") == 0) + { + build_ndistinct = true; + requested_type = true; + } + else if (strcmp(type, "dependencies") == 0) + { + build_dependencies = true; + requested_type = true; + } + else if (strcmp(type, "mcv") == 0) + { + build_mcv = true; + requested_type = true; + } + else ereport(ERROR, - (errcode(ERRCODE_TOO_MANY_COLUMNS), - errmsg("cannot have more than %d columns in statistics", - STATS_MAX_DIMENSIONS))); + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("unrecognized statistics kind \"%s\"", + type))); + } - attnums[numcols] = attForm->attnum; - numcols++; - ReleaseSysCache(atttuple); + /* + * If no statistic type was specified, build them all (but request + * expression stats only when there actually are any expressions). + */ + if (!requested_type) + { + build_ndistinct = (numcols >= 2); + build_dependencies = (numcols >= 2); + build_mcv = (numcols >= 2); } /* - * Check that at least two columns were specified in the statement. The - * upper bound was already checked in the loop above. + * When there are non-trivial expressions, build the expression stats + * automatically. This allows calculating good estimates for stats that + * consider per-clause estimates (e.g. functional dependencies). + */ + build_expressions = (list_length(stxexprs) > 0); + + /* + * Check that at least two columns were specified in the statement, or + * that we're building statistics on a single expression. */ - if (numcols < 2) + if ((numcols < 2) && (list_length(stxexprs) != 1)) ereport(ERROR, (errcode(ERRCODE_INVALID_OBJECT_DEFINITION), errmsg("extended statistics require at least 2 columns"))); @@ -265,13 +369,13 @@ CreateStatistics(CreateStatsStmt *stmt) * it does not hurt (it does not affect the efficiency, unlike for * indexes, for example). */ - qsort(attnums, numcols, sizeof(int16), compare_int16); + qsort(attnums, nattnums, sizeof(int16), compare_int16); /* * Check for duplicates in the list of columns. The attnums are sorted so * just check consecutive elements. */ - for (i = 1; i < numcols; i++) + for (i = 1; i < nattnums; i++) { if (attnums[i] == attnums[i - 1]) ereport(ERROR, @@ -279,48 +383,36 @@ CreateStatistics(CreateStatsStmt *stmt) errmsg("duplicate column name in statistics definition"))); } - /* Form an int2vector representation of the sorted column list */ - stxkeys = buildint2vector(attnums, numcols); - /* - * Parse the statistics kinds. + * Check for duplicate expressions. We do two loops, counting the + * occurrences of each expression. This is O(N^2) but we only allow + * small number of expressions and it's not executed often. */ - build_ndistinct = false; - build_dependencies = false; - build_mcv = false; - foreach(cell, stmt->stat_types) + foreach (cell, stxexprs) { - char *type = strVal((Value *) lfirst(cell)); + Node *expr1 = (Node *) lfirst(cell); + int cnt = 0; - if (strcmp(type, "ndistinct") == 0) - { - build_ndistinct = true; - requested_type = true; - } - else if (strcmp(type, "dependencies") == 0) + foreach (cell2, stxexprs) { - build_dependencies = true; - requested_type = true; - } - else if (strcmp(type, "mcv") == 0) - { - build_mcv = true; - requested_type = true; + Node *expr2 = (Node *) lfirst(cell2); + + if (equal(expr1, expr2)) + cnt += 1; } - else + + /* every expression should find at least itself */ + Assert(cnt >= 1); + + if (cnt > 1) ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("unrecognized statistics kind \"%s\"", - type))); - } - /* If no statistic type was specified, build them all. */ - if (!requested_type) - { - build_ndistinct = true; - build_dependencies = true; - build_mcv = true; + (errcode(ERRCODE_DUPLICATE_COLUMN), + errmsg("duplicate expression in statistics definition"))); } + /* Form an int2vector representation of the sorted column list */ + stxkeys = buildint2vector(attnums, nattnums); + /* construct the char array of enabled statistic types */ ntypes = 0; if (build_ndistinct) @@ -329,9 +421,23 @@ CreateStatistics(CreateStatsStmt *stmt) types[ntypes++] = CharGetDatum(STATS_EXT_DEPENDENCIES); if (build_mcv) types[ntypes++] = CharGetDatum(STATS_EXT_MCV); + if (build_expressions) + types[ntypes++] = CharGetDatum(STATS_EXT_EXPRESSIONS); Assert(ntypes > 0 && ntypes <= lengthof(types)); stxkind = construct_array(types, ntypes, CHAROID, 1, true, TYPALIGN_CHAR); + /* convert the expressions (if any) to a text datum */ + if (stxexprs != NIL) + { + char *exprsString; + + exprsString = nodeToString(stxexprs); + exprsDatum = CStringGetTextDatum(exprsString); + pfree(exprsString); + } + else + exprsDatum = (Datum) 0; + statrel = table_open(StatisticExtRelationId, RowExclusiveLock); /* @@ -351,6 +457,10 @@ CreateStatistics(CreateStatsStmt *stmt) values[Anum_pg_statistic_ext_stxkeys - 1] = PointerGetDatum(stxkeys); values[Anum_pg_statistic_ext_stxkind - 1] = PointerGetDatum(stxkind); + values[Anum_pg_statistic_ext_stxexprs - 1] = exprsDatum; + if (exprsDatum == (Datum) 0) + nulls[Anum_pg_statistic_ext_stxexprs - 1] = true; + /* insert it into pg_statistic_ext */ htup = heap_form_tuple(statrel->rd_att, values, nulls); CatalogTupleInsert(statrel, htup); @@ -373,6 +483,7 @@ CreateStatistics(CreateStatsStmt *stmt) datanulls[Anum_pg_statistic_ext_data_stxdndistinct - 1] = true; datanulls[Anum_pg_statistic_ext_data_stxddependencies - 1] = true; datanulls[Anum_pg_statistic_ext_data_stxdmcv - 1] = true; + datanulls[Anum_pg_statistic_ext_data_stxdexpr - 1] = true; /* insert it into pg_statistic_ext_data */ htup = heap_form_tuple(datarel->rd_att, datavalues, datanulls); @@ -396,12 +507,39 @@ CreateStatistics(CreateStatsStmt *stmt) */ ObjectAddressSet(myself, StatisticExtRelationId, statoid); - for (i = 0; i < numcols; i++) + /* add dependencies for plain column references */ + for (i = 0; i < nattnums; i++) { ObjectAddressSubSet(parentobject, RelationRelationId, relid, attnums[i]); recordDependencyOn(&myself, &parentobject, DEPENDENCY_AUTO); } + /* + * If there are no simply-referenced columns, give the statistics an + * auto dependency on the whole table. In most cases, this will + * be redundant, but it might not be if the statistics expressions + * contain no Vars (which might seem strange but possible). + * + * XXX This is copied from index_create, not sure if it's applicable + * to extended statistics too. + */ + if (!nattnums) + { + ObjectAddressSet(parentobject, RelationRelationId, relid); + recordDependencyOn(&myself, &parentobject, DEPENDENCY_AUTO); + } + + /* + * Store dependencies on anything mentioned in statistics expressions, + * just like we do for index expressions. + */ + if (stxexprs) + recordDependencyOnSingleRelExpr(&myself, + (Node *) stxexprs, + relid, + DEPENDENCY_NORMAL, + DEPENDENCY_AUTO, false, true); + /* * Also add dependencies on namespace and owner. These are required * because the stats object might have a different namespace and/or owner @@ -625,7 +763,7 @@ UpdateStatisticsForTypeChange(Oid statsOid, Oid relationOid, int attnum, elog(ERROR, "cache lookup failed for statistics object %u", statsOid); /* - * When none of the defined statistics types contain datum values from the + * When none of the defined statistics kinds contain datum values from the * table's columns then there's no need to reset the stats. Functional * dependencies and ndistinct stats should still hold true. */ @@ -637,7 +775,7 @@ UpdateStatisticsForTypeChange(Oid statsOid, Oid relationOid, int attnum, /* * OK, we need to reset some statistics. So let's build the new tuple, - * replacing the affected statistics types with NULL. + * replacing the affected statistics kinds with NULL. */ memset(nulls, 0, Natts_pg_statistic_ext_data * sizeof(bool)); memset(replaces, 0, Natts_pg_statistic_ext_data * sizeof(bool)); @@ -645,6 +783,7 @@ UpdateStatisticsForTypeChange(Oid statsOid, Oid relationOid, int attnum, replaces[Anum_pg_statistic_ext_data_stxdmcv - 1] = true; nulls[Anum_pg_statistic_ext_data_stxdmcv - 1] = true; + nulls[Anum_pg_statistic_ext_data_stxdexpr - 1] = true; rel = table_open(StatisticExtDataRelationId, RowExclusiveLock); @@ -731,18 +870,26 @@ ChooseExtendedStatisticNameAddition(List *exprs) buf[0] = '\0'; foreach(lc, exprs) { - ColumnRef *cref = (ColumnRef *) lfirst(lc); + StatsElem *selem = (StatsElem *) lfirst(lc); const char *name; /* It should be one of these, but just skip if it happens not to be */ - if (!IsA(cref, ColumnRef)) + if (!IsA(selem, StatsElem)) continue; - name = strVal((Value *) linitial(cref->fields)); + name = selem->name; if (buflen > 0) buf[buflen++] = '_'; /* insert _ between names */ + /* + * FIXME use 'expr' for expressions, which have empty column names. + * For indexes this is handled in ChooseIndexColumnNames, but we + * have no such function for stats. + */ + if (!name) + name = "expr"; + /* * At this point we have buflen <= NAMEDATALEN. name should be less * than NAMEDATALEN already, but use strlcpy for paranoia. diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index aaba1ec2c4..e3779f0702 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2980,6 +2980,17 @@ _copyIndexElem(const IndexElem *from) return newnode; } +static StatsElem * +_copyStatsElem(const StatsElem *from) +{ + StatsElem *newnode = makeNode(StatsElem); + + COPY_STRING_FIELD(name); + COPY_NODE_FIELD(expr); + + return newnode; +} + static ColumnDef * _copyColumnDef(const ColumnDef *from) { @@ -5695,6 +5706,9 @@ copyObjectImpl(const void *from) case T_IndexElem: retval = _copyIndexElem(from); break; + case T_StatsElem: + retval = _copyStatsElem(from); + break; case T_ColumnDef: retval = _copyColumnDef(from); break; diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..1c743b7539 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2594,6 +2594,16 @@ _equalIndexElem(const IndexElem *a, const IndexElem *b) return true; } + +static bool +_equalStatsElem(const StatsElem *a, const StatsElem *b) +{ + COMPARE_STRING_FIELD(name); + COMPARE_NODE_FIELD(expr); + + return true; +} + static bool _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { @@ -3720,6 +3730,9 @@ equal(const void *a, const void *b) case T_IndexElem: retval = _equalIndexElem(a, b); break; + case T_StatsElem: + retval = _equalStatsElem(a, b); + break; case T_ColumnDef: retval = _equalColumnDef(a, b); break; diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 8fc432bfe1..4142ecc1c7 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2941,6 +2941,15 @@ _outIndexElem(StringInfo str, const IndexElem *node) WRITE_ENUM_FIELD(nulls_ordering, SortByNulls); } +static void +_outStatsElem(StringInfo str, const StatsElem *node) +{ + WRITE_NODE_TYPE("STATSELEM"); + + WRITE_STRING_FIELD(name); + WRITE_NODE_FIELD(expr); +} + static void _outQuery(StringInfo str, const Query *node) { @@ -4283,6 +4292,9 @@ outNode(StringInfo str, const void *obj) case T_IndexElem: _outIndexElem(str, obj); break; + case T_StatsElem: + _outStatsElem(str, obj); + break; case T_Query: _outQuery(str, obj); break; diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index c5947fa418..3901edf154 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -34,6 +34,7 @@ #include "foreign/fdwapi.h" #include "miscadmin.h" #include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" #include "nodes/supportnodes.h" #include "optimizer/clauses.h" #include "optimizer/cost.h" @@ -1309,6 +1310,7 @@ get_relation_constraints(PlannerInfo *root, static List * get_relation_statistics(RelOptInfo *rel, Relation relation) { + Index varno = rel->relid; List *statoidlist; List *stainfos = NIL; ListCell *l; @@ -1323,6 +1325,7 @@ get_relation_statistics(RelOptInfo *rel, Relation relation) HeapTuple dtup; Bitmapset *keys = NULL; int i; + List *exprs = NIL; htup = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(statOid)); if (!HeapTupleIsValid(htup)) @@ -1341,6 +1344,49 @@ get_relation_statistics(RelOptInfo *rel, Relation relation) for (i = 0; i < staForm->stxkeys.dim1; i++) keys = bms_add_member(keys, staForm->stxkeys.values[i]); + /* + * preprocess expression (if any) + * + * FIXME Should we cache the result somewhere? + */ + { + bool isnull; + Datum datum; + + /* decode expression (if any) */ + datum = SysCacheGetAttr(STATEXTOID, htup, + Anum_pg_statistic_ext_stxexprs, &isnull); + + if (!isnull) + { + char *exprsString; + + exprsString = TextDatumGetCString(datum); + exprs = (List *) stringToNode(exprsString); + pfree(exprsString); + + /* + * Run the expressions through eval_const_expressions. This is not just an + * optimization, but is necessary, because the planner will be comparing + * them to similarly-processed qual clauses, and may fail to detect valid + * matches without this. We must not use canonicalize_qual, however, + * since these aren't qual expressions. + */ + exprs = (List *) eval_const_expressions(NULL, (Node *) exprs); + + /* May as well fix opfuncids too */ + fix_opfuncids((Node *) exprs); + + /* + * Modify the copies we obtain from the relcache to have the + * correct varno for the parent relation, so that they match up + * correctly against qual clauses. + */ + if (varno != 1) + ChangeVarNodes((Node *) exprs, 1, varno, 0); + } + } + /* add one StatisticExtInfo for each kind built */ if (statext_is_kind_built(dtup, STATS_EXT_NDISTINCT)) { @@ -1350,6 +1396,7 @@ get_relation_statistics(RelOptInfo *rel, Relation relation) info->rel = rel; info->kind = STATS_EXT_NDISTINCT; info->keys = bms_copy(keys); + info->exprs = exprs; stainfos = lappend(stainfos, info); } @@ -1362,6 +1409,7 @@ get_relation_statistics(RelOptInfo *rel, Relation relation) info->rel = rel; info->kind = STATS_EXT_DEPENDENCIES; info->keys = bms_copy(keys); + info->exprs = exprs; stainfos = lappend(stainfos, info); } @@ -1374,6 +1422,20 @@ get_relation_statistics(RelOptInfo *rel, Relation relation) info->rel = rel; info->kind = STATS_EXT_MCV; info->keys = bms_copy(keys); + info->exprs = exprs; + + stainfos = lappend(stainfos, info); + } + + if (statext_is_kind_built(dtup, STATS_EXT_EXPRESSIONS)) + { + StatisticExtInfo *info = makeNode(StatisticExtInfo); + + info->statOid = statOid; + info->rel = rel; + info->kind = STATS_EXT_EXPRESSIONS; + info->keys = bms_copy(keys); + info->exprs = exprs; stainfos = lappend(stainfos, info); } diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..fe0b6d7c54 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -232,6 +232,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); WindowDef *windef; JoinExpr *jexpr; IndexElem *ielem; + StatsElem *selem; Alias *alias; RangeVar *range; IntoClause *into; @@ -396,7 +397,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); old_aggr_definition old_aggr_list oper_argtypes RuleActionList RuleActionMulti opt_column_list columnList opt_name_list - sort_clause opt_sort_clause sortby_list index_params + sort_clause opt_sort_clause sortby_list index_params stats_params opt_include opt_c_include index_including_params name_list role_list from_clause from_list opt_array_bounds qualified_name_list any_name any_name_list type_name_list @@ -502,6 +503,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <list> func_alias_clause %type <sortby> sortby %type <ielem> index_elem index_elem_options +%type <selem> stats_param %type <node> table_ref %type <jexpr> joined_table %type <range> relation_expr @@ -4051,7 +4053,7 @@ ExistingIndex: USING INDEX name { $$ = $3; } CreateStatsStmt: CREATE STATISTICS any_name - opt_name_list ON expr_list FROM from_list + opt_name_list ON stats_params FROM from_list { CreateStatsStmt *n = makeNode(CreateStatsStmt); n->defnames = $3; @@ -4063,7 +4065,7 @@ CreateStatsStmt: $$ = (Node *)n; } | CREATE STATISTICS IF_P NOT EXISTS any_name - opt_name_list ON expr_list FROM from_list + opt_name_list ON stats_params FROM from_list { CreateStatsStmt *n = makeNode(CreateStatsStmt); n->defnames = $6; @@ -4076,6 +4078,36 @@ CreateStatsStmt: } ; +/* + * Statistics attributes can be either simple column references, or arbitrary + * expressions in parens. For compatibility with index attributes permitted + * in CREATE INDEX, we allow an expression that's just a function call to be + * written without parens. + */ + +stats_params: stats_param { $$ = list_make1($1); } + | stats_params ',' stats_param { $$ = lappend($1, $3); } + ; + +stats_param: ColId + { + $$ = makeNode(StatsElem); + $$->name = $1; + $$->expr = NULL; + } + | func_expr_windowless + { + $$ = makeNode(StatsElem); + $$->name = NULL; + $$->expr = $1; + } + | '(' a_expr ')' + { + $$ = makeNode(StatsElem); + $$->name = NULL; + $$->expr = $2; + } + ; /***************************************************************************** * diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index fd08b9eeff..1dea9a7616 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -484,6 +484,13 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) else err = _("grouping operations are not allowed in index predicates"); + break; + case EXPR_KIND_STATS_EXPRESSION: + if (isAgg) + err = _("aggregate functions are not allowed in statistics expressions"); + else + err = _("grouping operations are not allowed in statistics expressions"); + break; case EXPR_KIND_ALTER_COL_TRANSFORM: if (isAgg) @@ -910,6 +917,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_INDEX_EXPRESSION: err = _("window functions are not allowed in index expressions"); break; + case EXPR_KIND_STATS_EXPRESSION: + err = _("window functions are not allowed in statistics expressions"); + break; case EXPR_KIND_INDEX_PREDICATE: err = _("window functions are not allowed in index predicates"); break; diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index f869e159d6..03373d551f 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -500,6 +500,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_FUNCTION_DEFAULT: case EXPR_KIND_INDEX_EXPRESSION: case EXPR_KIND_INDEX_PREDICATE: + case EXPR_KIND_STATS_EXPRESSION: case EXPR_KIND_ALTER_COL_TRANSFORM: case EXPR_KIND_EXECUTE_PARAMETER: case EXPR_KIND_TRIGGER_WHEN: @@ -1741,6 +1742,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_INDEX_PREDICATE: err = _("cannot use subquery in index predicate"); break; + case EXPR_KIND_STATS_EXPRESSION: + err = _("cannot use subquery in statistics expression"); + break; case EXPR_KIND_ALTER_COL_TRANSFORM: err = _("cannot use subquery in transform expression"); break; @@ -3030,6 +3034,8 @@ ParseExprKindName(ParseExprKind exprKind) return "index expression"; case EXPR_KIND_INDEX_PREDICATE: return "index predicate"; + case EXPR_KIND_STATS_EXPRESSION: + return "statistics expression"; case EXPR_KIND_ALTER_COL_TRANSFORM: return "USING"; case EXPR_KIND_EXECUTE_PARAMETER: diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 37cebc7d82..debef1d14f 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2503,6 +2503,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_INDEX_PREDICATE: err = _("set-returning functions are not allowed in index predicates"); break; + case EXPR_KIND_STATS_EXPRESSION: + err = _("set-returning functions are not allowed in statistics expressions"); + break; case EXPR_KIND_ALTER_COL_TRANSFORM: err = _("set-returning functions are not allowed in transform expressions"); break; diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..8830f351eb 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -1898,6 +1898,9 @@ generateClonedExtStatsStmt(RangeVar *heapRel, Oid heapRelid, stat_types = lappend(stat_types, makeString("dependencies")); else if (enabled[i] == STATS_EXT_MCV) stat_types = lappend(stat_types, makeString("mcv")); + else if (enabled[i] == STATS_EXT_EXPRESSIONS) + /* expression stats are not exposed to users */ + continue; else elog(ERROR, "unrecognized statistics kind %c", enabled[i]); } @@ -1905,14 +1908,43 @@ generateClonedExtStatsStmt(RangeVar *heapRel, Oid heapRelid, /* Determine which columns the statistics are on */ for (i = 0; i < statsrec->stxkeys.dim1; i++) { - ColumnRef *cref = makeNode(ColumnRef); + StatsElem *selem = makeNode(StatsElem); AttrNumber attnum = statsrec->stxkeys.values[i]; - cref->fields = list_make1(makeString(get_attname(heapRelid, - attnum, false))); - cref->location = -1; + selem->name = get_attname(heapRelid, attnum, false); + selem->expr = NULL; - def_names = lappend(def_names, cref); + def_names = lappend(def_names, selem); + } + + /* + * Now handle expressions, if there are any. The order does not + * matter for extended stats, so we simply append them after + * simple column references. + */ + datum = SysCacheGetAttr(STATEXTOID, ht_stats, + Anum_pg_statistic_ext_stxexprs, &isnull); + + if (!isnull) + { + ListCell *lc; + List *exprs = NIL; + char *exprsString; + + exprsString = TextDatumGetCString(datum); + exprs = (List *) stringToNode(exprsString); + + foreach(lc, exprs) + { + StatsElem *selem = makeNode(StatsElem); + + selem->name = NULL; + selem->expr = (Node *) lfirst(lc); + + def_names = lappend(def_names, selem); + } + + pfree(exprsString); } /* finally, build the output node */ @@ -1923,6 +1955,7 @@ generateClonedExtStatsStmt(RangeVar *heapRel, Oid heapRelid, stats->relations = list_make1(heapRel); stats->stxcomment = NULL; stats->if_not_exists = false; + stats->transformed = true; /* don't need transformStatsStmt */ /* Clean up */ ReleaseSysCache(ht_stats); @@ -2847,6 +2880,84 @@ transformIndexStmt(Oid relid, IndexStmt *stmt, const char *queryString) return stmt; } +/* + * transformStatsStmt - parse analysis for CREATE STATISTICS + * + * To avoid race conditions, it's important that this function rely only on + * the passed-in relid (and not on stmt->relation) to determine the target + * relation. + */ +CreateStatsStmt * +transformStatsStmt(Oid relid, CreateStatsStmt *stmt, const char *queryString) +{ + ParseState *pstate; + ParseNamespaceItem *nsitem; + ListCell *l; + Relation rel; + + /* Nothing to do if statement already transformed. */ + if (stmt->transformed) + return stmt; + + /* + * We must not scribble on the passed-in CreateStatsStmt, so copy it. (This is + * overkill, but easy.) + */ + stmt = copyObject(stmt); + + /* Set up pstate */ + pstate = make_parsestate(NULL); + pstate->p_sourcetext = queryString; + + /* + * Put the parent table into the rtable so that the expressions can refer + * to its fields without qualification. Caller is responsible for locking + * relation, but we still need to open it. + */ + rel = relation_open(relid, NoLock); + nsitem = addRangeTableEntryForRelation(pstate, rel, + AccessShareLock, + NULL, false, true); + + /* no to join list, yes to namespaces */ + addNSItemToQuery(pstate, nsitem, false, true, true); + + /* take care of any expressions */ + foreach(l, stmt->exprs) + { + StatsElem *selem = (StatsElem *) lfirst(l); + + if (selem->expr) + { + /* Now do parse transformation of the expression */ + selem->expr = transformExpr(pstate, selem->expr, + EXPR_KIND_STATS_EXPRESSION); + + /* We have to fix its collations too */ + assign_expr_collations(pstate, selem->expr); + } + } + + /* + * Check that only the base rel is mentioned. (This should be dead code + * now that add_missing_from is history.) + */ + if (list_length(pstate->p_rtable) != 1) + ereport(ERROR, + (errcode(ERRCODE_INVALID_COLUMN_REFERENCE), + errmsg("statistics expressions and predicates can refer only to the table being indexed"))); + + free_parsestate(pstate); + + /* Close relation */ + table_close(rel, NoLock); + + /* Mark statement as successfully transformed */ + stmt->transformed = true; + + return stmt; +} + /* * transformRuleStmt - diff --git a/src/backend/statistics/dependencies.c b/src/backend/statistics/dependencies.c index f6e399b192..6bf3127bcc 100644 --- a/src/backend/statistics/dependencies.c +++ b/src/backend/statistics/dependencies.c @@ -70,15 +70,18 @@ static void generate_dependencies(DependencyGenerator state); static DependencyGenerator DependencyGenerator_init(int n, int k); static void DependencyGenerator_free(DependencyGenerator state); static AttrNumber *DependencyGenerator_next(DependencyGenerator state); -static double dependency_degree(int numrows, HeapTuple *rows, int k, - AttrNumber *dependency, VacAttrStats **stats, Bitmapset *attrs); +static double dependency_degree(int numrows, HeapTuple *rows, + ExprInfo *exprs, int k, + AttrNumber *dependency, VacAttrStats **stats, + Bitmapset *attrs); static bool dependency_is_fully_matched(MVDependency *dependency, Bitmapset *attnums); static bool dependency_is_compatible_clause(Node *clause, Index relid, AttrNumber *attnum); +static bool dependency_is_compatible_expression(Node *clause, Index relid, + List *statlist, Node **expr); static MVDependency *find_strongest_dependency(MVDependencies **dependencies, - int ndependencies, - Bitmapset *attnums); + int ndependencies, Bitmapset *attnums); static Selectivity clauselist_apply_dependencies(PlannerInfo *root, List *clauses, int varRelid, JoinType jointype, SpecialJoinInfo *sjinfo, @@ -219,8 +222,9 @@ DependencyGenerator_next(DependencyGenerator state) * the last one. */ static double -dependency_degree(int numrows, HeapTuple *rows, int k, AttrNumber *dependency, - VacAttrStats **stats, Bitmapset *attrs) +dependency_degree(int numrows, HeapTuple *rows, ExprInfo *exprs, int k, + AttrNumber *dependency, VacAttrStats **stats, + Bitmapset *attrs) { int i, nitems; @@ -289,8 +293,8 @@ dependency_degree(int numrows, HeapTuple *rows, int k, AttrNumber *dependency, * descriptor. For now that assumption holds, but it might change in the * future for example if we support statistics on multiple tables. */ - items = build_sorted_items(numrows, &nitems, rows, stats[0]->tupDesc, - mss, k, attnums_dep); + items = build_sorted_items(numrows, &nitems, rows, exprs, + stats[0]->tupDesc, mss, k, attnums_dep); /* * Walk through the sorted array, split it into rows according to the @@ -360,7 +364,8 @@ dependency_degree(int numrows, HeapTuple *rows, int k, AttrNumber *dependency, * (c) -> b */ MVDependencies * -statext_dependencies_build(int numrows, HeapTuple *rows, Bitmapset *attrs, +statext_dependencies_build(int numrows, HeapTuple *rows, + ExprInfo *exprs, Bitmapset *attrs, VacAttrStats **stats) { int i, @@ -371,6 +376,9 @@ statext_dependencies_build(int numrows, HeapTuple *rows, Bitmapset *attrs, /* result */ MVDependencies *dependencies = NULL; + /* treat expressions as special attributes with high attnums */ + attrs = add_expressions_to_attributes(attrs, exprs->nexprs); + /* * Transform the bms into an array, to make accessing i-th member easier. */ @@ -398,7 +406,8 @@ statext_dependencies_build(int numrows, HeapTuple *rows, Bitmapset *attrs, MVDependency *d; /* compute how valid the dependency seems */ - degree = dependency_degree(numrows, rows, k, dependency, stats, attrs); + degree = dependency_degree(numrows, rows, exprs, k, dependency, + stats, attrs); /* * if the dependency seems entirely invalid, don't store it @@ -441,6 +450,8 @@ statext_dependencies_build(int numrows, HeapTuple *rows, Bitmapset *attrs, DependencyGenerator_free(DependencyGenerator); } + pfree(attrs); + return dependencies; } @@ -639,7 +650,7 @@ statext_dependencies_load(Oid mvoid) Anum_pg_statistic_ext_data_stxddependencies, &isnull); if (isnull) elog(ERROR, - "requested statistic kind \"%c\" is not yet built for statistics object %u", + "requested statistics kind \"%c\" is not yet built for statistics object %u", STATS_EXT_DEPENDENCIES, mvoid); result = statext_dependencies_deserialize(DatumGetByteaPP(deps)); @@ -1157,6 +1168,134 @@ clauselist_apply_dependencies(PlannerInfo *root, List *clauses, return s1; } +/* + * dependency_is_compatible_expression + * Determines if the expression is compatible with functional dependencies + * + * Similar to dependency_is_compatible_clause, but doesn't enforce that the + * expression is a simple Var. OTOH we check that there's at least one + * statistics object matching the expression. + */ +static bool +dependency_is_compatible_expression(Node *clause, Index relid, List *statlist, Node **expr) +{ + List *vars; + ListCell *lc, *lc2; + + RestrictInfo *rinfo = (RestrictInfo *) clause; + Node *clause_expr; + + if (!IsA(rinfo, RestrictInfo)) + return false; + + /* Pseudoconstants are not interesting (they couldn't contain a Var) */ + if (rinfo->pseudoconstant) + return false; + + /* Clauses referencing multiple, or no, varnos are incompatible */ + if (bms_membership(rinfo->clause_relids) != BMS_SINGLETON) + return false; + + if (is_opclause(rinfo->clause)) + { + /* If it's an opclause, check for Var = Const or Const = Var. */ + OpExpr *expr = (OpExpr *) rinfo->clause; + + /* Only expressions with two arguments are candidates. */ + if (list_length(expr->args) != 2) + return false; + + /* Make sure non-selected argument is a pseudoconstant. */ + if (is_pseudo_constant_clause(lsecond(expr->args))) + clause_expr = linitial(expr->args); + else if (is_pseudo_constant_clause(linitial(expr->args))) + clause_expr = lsecond(expr->args); + else + return false; + + /* + * If it's not an "=" operator, just ignore the clause, as it's not + * compatible with functional dependencies. + * + * This uses the function for estimating selectivity, not the operator + * directly (a bit awkward, but well ...). + * + * XXX this is pretty dubious; probably it'd be better to check btree + * or hash opclass membership, so as not to be fooled by custom + * selectivity functions, and to be more consistent with decisions + * elsewhere in the planner. + */ + if (get_oprrest(expr->opno) != F_EQSEL) + return false; + + /* OK to proceed with checking "var" */ + } + else if (is_notclause(rinfo->clause)) + { + /* + * "NOT x" can be interpreted as "x = false", so get the argument and + * proceed with seeing if it's a suitable Var. + */ + clause_expr = (Node *) get_notclausearg(rinfo->clause); + } + else + { + /* + * A boolean expression "x" can be interpreted as "x = true", so + * proceed with seeing if it's a suitable Var. + */ + clause_expr = (Node *) rinfo->clause; + } + + /* + * We may ignore any RelabelType node above the operand. (There won't be + * more than one, since eval_const_expressions has been applied already.) + */ + if (IsA(clause_expr, RelabelType)) + clause_expr = (Node *) ((RelabelType *) clause_expr)->arg; + + vars = pull_var_clause(clause_expr, 0); + + foreach (lc, vars) + { + Var *var = (Var *) lfirst(lc); + + /* Ensure Var is from the correct relation */ + if (var->varno != relid) + return false; + + /* We also better ensure the Var is from the current level */ + if (var->varlevelsup != 0) + return false; + + /* Also ignore system attributes (we don't allow stats on those) */ + if (!AttrNumberIsForUserDefinedAttr(var->varattno)) + return false; + } + + foreach (lc, statlist) + { + StatisticExtInfo *info = (StatisticExtInfo *) lfirst(lc); + + /* ignore stats without dependencies */ + if (info->kind != STATS_EXT_DEPENDENCIES) + continue; + + foreach (lc2, info->exprs) + { + Node *stat_expr = (Node *) lfirst(lc2); + + if (equal(clause_expr, stat_expr)) + { + *expr = stat_expr; + return true; + } + } + } + + return false; +} + /* * dependencies_clauselist_selectivity * Return the estimated selectivity of (a subset of) the given clauses @@ -1205,6 +1344,10 @@ dependencies_clauselist_selectivity(PlannerInfo *root, int ndependencies; int i; + /* unique expressions */ + Node **unique_exprs; + int unique_exprs_cnt; + /* check if there's any stats that might be useful for us. */ if (!has_stats_of_kind(rel->statlist, STATS_EXT_DEPENDENCIES)) return 1.0; @@ -1212,6 +1355,14 @@ dependencies_clauselist_selectivity(PlannerInfo *root, list_attnums = (AttrNumber *) palloc(sizeof(AttrNumber) * list_length(clauses)); + /* + * We allocate space as if every clause was a unique expression, although + * that's probably overkill. Some will be simple column references that + * we'll translate to attnums, and there might be duplicates. + */ + unique_exprs = (Node **) palloc(sizeof(Node *) * list_length(clauses)); + unique_exprs_cnt = 0; + /* * Pre-process the clauses list to extract the attnums seen in each item. * We need to determine if there's any clauses which will be useful for @@ -1222,29 +1373,76 @@ dependencies_clauselist_selectivity(PlannerInfo *root, * * We also skip clauses that we already estimated using different types of * statistics (we treat them as incompatible). + * + * For expressions, we generate attnums higher than MaxHeapAttributeNumber + * so that we can work with attnums only. */ listidx = 0; foreach(l, clauses) { Node *clause = (Node *) lfirst(l); AttrNumber attnum; + Node *expr = NULL; - if (!bms_is_member(listidx, *estimatedclauses) && - dependency_is_compatible_clause(clause, rel->relid, &attnum)) + /* ignore clause by default */ + list_attnums[listidx] = InvalidAttrNumber; + + if (!bms_is_member(listidx, *estimatedclauses)) { - list_attnums[listidx] = attnum; - clauses_attnums = bms_add_member(clauses_attnums, attnum); + /* + * If it's a simple column refrence, just extract the attnum. If + * it's an expression, make sure it's not a duplicate and assign + * a special attnum to it (higher than any regular value). + */ + if (dependency_is_compatible_clause(clause, rel->relid, &attnum)) + { + list_attnums[listidx] = attnum; + clauses_attnums = bms_add_member(clauses_attnums, attnum); + } + else if (dependency_is_compatible_expression(clause, rel->relid, + rel->statlist, + &expr)) + { + /* special attnum assigned to this expression */ + attnum = InvalidAttrNumber; + + Assert(expr != NULL); + + /* If the expression is duplicate, use the same attnum. */ + for (i = 0; i < unique_exprs_cnt; i++) + { + if (equal(unique_exprs[i], expr)) + { + attnum = EXPRESSION_ATTNUM(i); + break; + } + } + + /* not found in the list, so add it */ + if (attnum == InvalidAttrNumber) + { + attnum = EXPRESSION_ATTNUM(unique_exprs_cnt); + unique_exprs[unique_exprs_cnt++] = expr; + + /* shouldn't have seen this attnum yet */ + Assert(!bms_is_member(attnum, clauses_attnums)); + + /* we may add the attnum repeatedly to clauses_attnums */ + clauses_attnums = bms_add_member(clauses_attnums, attnum); + } + + /* remember which attnum was assigned to this clause */ + list_attnums[listidx] = attnum; + } } - else - list_attnums[listidx] = InvalidAttrNumber; listidx++; } /* - * If there's not at least two distinct attnums then reject the whole list - * of clauses. We must return 1.0 so the calling function's selectivity is - * unaffected. + * If there's not at least two distinct attnums and expressions, then + * reject the whole list of clauses. We must return 1.0 so the calling + * function's selectivity is unaffected. */ if (bms_membership(clauses_attnums) != BMS_MULTIPLE) { @@ -1273,25 +1471,138 @@ dependencies_clauselist_selectivity(PlannerInfo *root, { StatisticExtInfo *stat = (StatisticExtInfo *) lfirst(l); Bitmapset *matched; - BMS_Membership membership; + int nmatched; + int nexprs; + MVDependencies *deps; /* skip statistics that are not of the correct type */ if (stat->kind != STATS_EXT_DEPENDENCIES) continue; + /* count matching simple clauses */ matched = bms_intersect(clauses_attnums, stat->keys); - membership = bms_membership(matched); + nmatched = bms_num_members(matched); bms_free(matched); - /* skip objects matching fewer than two attributes from clauses */ - if (membership != BMS_MULTIPLE) + /* count matching expressions */ + nexprs = 0; + for (i = 0; i < unique_exprs_cnt; i++) + { + ListCell *lc; + + foreach (lc, stat->exprs) + { + Node *stat_expr = (Node *) lfirst(lc); + + /* try to match it */ + if (equal(stat_expr, unique_exprs[i])) + nexprs++; + } + } + + /* + * Skip objects matching fewer than two attributes/expressions + * from clauses. + */ + if (nmatched + nexprs < 2) continue; - func_dependencies[nfunc_dependencies] - = statext_dependencies_load(stat->statOid); + deps = statext_dependencies_load(stat->statOid); - total_ndeps += func_dependencies[nfunc_dependencies]->ndeps; - nfunc_dependencies++; + /* + * The expressions may be represented by different attnums in the + * stats, we need to remap them to be consistent with the clauses. + * That will make the later steps (e.g. picking the strongest item + * and so on) much simpler. + * + * When we're at it, we can also remove dependencies referencing + * missing clauses (i.e. expressions that are not in the clauses). + * + * XXX We might also skip clauses referencing missing attnums, not + * just expressions. + */ + if (stat->exprs) + { + int ndeps = 0; + + for (i = 0; i < deps->ndeps; i++) + { + bool skip = false; + MVDependency *dep = deps->deps[i]; + int j; + + for (j = 0; j < dep->nattributes; j++) + { + int idx; + Node *expr; + int k; + AttrNumber unique_attnum = InvalidAttrNumber; + + /* regular attribute, no need to remap */ + if (dep->attributes[j] <= MaxHeapAttributeNumber) + continue; + + /* index of the expression */ + idx = EXPRESSION_INDEX(dep->attributes[j]); + + /* make sure the expression index is valid */ + Assert((idx >= 0) && (idx < list_length(stat->exprs))); + + expr = (Node *) list_nth(stat->exprs, idx); + + /* try to find the expression in the unique list */ + for (k = 0; k < unique_exprs_cnt; k++) + { + /* + * found a matching unique expression, use the attnum + * (derived from index of the unique expression) + */ + if (equal(unique_exprs[k], expr)) + { + unique_attnum = EXPRESSION_ATTNUM(k); + break; + } + } + + /* + * Not found a matching expression, so we can simply + * skip this dependency, because there's no chance it + * will be fully covered. + */ + if (unique_attnum == InvalidAttrNumber) + { + skip = true; + break; + } + + /* otherwise remap it to the new attnum */ + dep->attributes[j] = unique_attnum; + } + + /* if found a matching, */ + if (!skip) + { + /* maybe we've skipped something earlier, so move it */ + if (ndeps != i) + deps->deps[ndeps] = deps->deps[i]; + + ndeps++; + } + } + + deps->ndeps = ndeps; + } + + /* + * It's possible we've removed all dependencies, in which case we + * don't bother adding it to the list. + */ + if (deps->ndeps > 0) + { + func_dependencies[nfunc_dependencies] = deps; + total_ndeps += deps->ndeps; + nfunc_dependencies++; + } } /* if no matching stats could be found then we've nothing to do */ @@ -1300,6 +1611,7 @@ dependencies_clauselist_selectivity(PlannerInfo *root, pfree(func_dependencies); bms_free(clauses_attnums); pfree(list_attnums); + pfree(unique_exprs); return 1.0; } @@ -1347,6 +1659,7 @@ dependencies_clauselist_selectivity(PlannerInfo *root, pfree(func_dependencies); bms_free(clauses_attnums); pfree(list_attnums); + pfree(unique_exprs); return s1; } diff --git a/src/backend/statistics/extended_stats.c b/src/backend/statistics/extended_stats.c index a030ea3653..6ed938d6ab 100644 --- a/src/backend/statistics/extended_stats.c +++ b/src/backend/statistics/extended_stats.c @@ -24,6 +24,7 @@ #include "catalog/pg_collation.h" #include "catalog/pg_statistic_ext.h" #include "catalog/pg_statistic_ext_data.h" +#include "executor/executor.h" #include "commands/progress.h" #include "miscadmin.h" #include "nodes/nodeFuncs.h" @@ -35,13 +36,16 @@ #include "statistics/statistics.h" #include "utils/acl.h" #include "utils/array.h" +#include "utils/attoptcache.h" #include "utils/builtins.h" +#include "utils/datum.h" #include "utils/fmgroids.h" #include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/rel.h" #include "utils/selfuncs.h" #include "utils/syscache.h" +#include "utils/typcache.h" /* * To avoid consuming too much memory during analysis and/or too much space @@ -64,20 +68,37 @@ typedef struct StatExtEntry char *schema; /* statistics object's schema */ char *name; /* statistics object's name */ Bitmapset *columns; /* attribute numbers covered by the object */ - List *types; /* 'char' list of enabled statistic kinds */ + List *types; /* 'char' list of enabled statistics kinds */ int stattarget; /* statistics target (-1 for default) */ + List *exprs; /* expressions */ } StatExtEntry; static List *fetch_statentries_for_relation(Relation pg_statext, Oid relid); -static VacAttrStats **lookup_var_attr_stats(Relation rel, Bitmapset *attrs, +static VacAttrStats **lookup_var_attr_stats(Relation rel, Bitmapset *attrs, List *exprs, int nvacatts, VacAttrStats **vacatts); -static void statext_store(Oid relid, +static void statext_store(Oid statOid, MVNDistinct *ndistinct, MVDependencies *dependencies, - MCVList *mcv, VacAttrStats **stats); + MCVList *mcv, Datum exprs, VacAttrStats **stats); static int statext_compute_stattarget(int stattarget, int natts, VacAttrStats **stats); +typedef struct AnlExprData +{ + Node *expr; /* expression to analyze */ + VacAttrStats *vacattrstat; /* index attrs to analyze */ +} AnlExprData; + +static void compute_expr_stats(Relation onerel, double totalrows, + AnlExprData *exprdata, int nexprs, + HeapTuple *rows, int numrows); +static Datum serialize_expr_stats(AnlExprData *exprdata, int nexprs); +static Datum expr_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull); +static AnlExprData *build_expr_data(List *exprs); +static VacAttrStats *examine_expression(Node *expr); +static ExprInfo *evaluate_expressions(Relation rel, List *exprs, + int numrows, HeapTuple *rows); + /* * Compute requested extended stats, using the rows sampled for the plain * (single-column) stats. @@ -92,21 +113,25 @@ BuildRelationExtStatistics(Relation onerel, double totalrows, { Relation pg_stext; ListCell *lc; - List *stats; + List *statslist; MemoryContext cxt; MemoryContext oldcxt; int64 ext_cnt; + /* Do nothing if there are no columns to analyze. */ + if (!natts) + return; + cxt = AllocSetContextCreate(CurrentMemoryContext, "BuildRelationExtStatistics", ALLOCSET_DEFAULT_SIZES); oldcxt = MemoryContextSwitchTo(cxt); pg_stext = table_open(StatisticExtRelationId, RowExclusiveLock); - stats = fetch_statentries_for_relation(pg_stext, RelationGetRelid(onerel)); + statslist = fetch_statentries_for_relation(pg_stext, RelationGetRelid(onerel)); /* report this phase */ - if (stats != NIL) + if (statslist != NIL) { const int index[] = { PROGRESS_ANALYZE_PHASE, @@ -114,28 +139,30 @@ BuildRelationExtStatistics(Relation onerel, double totalrows, }; const int64 val[] = { PROGRESS_ANALYZE_PHASE_COMPUTE_EXT_STATS, - list_length(stats) + list_length(statslist) }; pgstat_progress_update_multi_param(2, index, val); } ext_cnt = 0; - foreach(lc, stats) + foreach(lc, statslist) { StatExtEntry *stat = (StatExtEntry *) lfirst(lc); MVNDistinct *ndistinct = NULL; MVDependencies *dependencies = NULL; MCVList *mcv = NULL; + Datum exprstats = (Datum) 0; VacAttrStats **stats; ListCell *lc2; int stattarget; + ExprInfo *exprs; /* * Check if we can build these stats based on the column analyzed. If * not, report this fact (except in autovacuum) and move on. */ - stats = lookup_var_attr_stats(onerel, stat->columns, + stats = lookup_var_attr_stats(onerel, stat->columns, stat->exprs, natts, vacattrstats); if (!stats) { @@ -150,10 +177,6 @@ BuildRelationExtStatistics(Relation onerel, double totalrows, continue; } - /* check allowed number of dimensions */ - Assert(bms_num_members(stat->columns) >= 2 && - bms_num_members(stat->columns) <= STATS_MAX_DIMENSIONS); - /* compute statistics target for this statistics */ stattarget = statext_compute_stattarget(stat->stattarget, bms_num_members(stat->columns), @@ -167,6 +190,9 @@ BuildRelationExtStatistics(Relation onerel, double totalrows, if (stattarget == 0) continue; + /* evaluate expressions (if the statistics has any) */ + exprs = evaluate_expressions(onerel, stat->exprs, numrows, rows); + /* compute statistic of each requested type */ foreach(lc2, stat->types) { @@ -174,21 +200,43 @@ BuildRelationExtStatistics(Relation onerel, double totalrows, if (t == STATS_EXT_NDISTINCT) ndistinct = statext_ndistinct_build(totalrows, numrows, rows, - stat->columns, stats); + exprs, stat->columns, + stats); else if (t == STATS_EXT_DEPENDENCIES) dependencies = statext_dependencies_build(numrows, rows, - stat->columns, stats); + exprs, stat->columns, + stats); else if (t == STATS_EXT_MCV) - mcv = statext_mcv_build(numrows, rows, stat->columns, stats, - totalrows, stattarget); + mcv = statext_mcv_build(numrows, rows, exprs, stat->columns, + stats, totalrows, stattarget); + else if (t == STATS_EXT_EXPRESSIONS) + { + AnlExprData *exprdata; + int nexprs; + + /* should not happen, thanks to checks when defining stats */ + if (!stat->exprs) + elog(ERROR, "requested expression stats, but there are no expressions"); + + exprdata = build_expr_data(stat->exprs); + nexprs = list_length(stat->exprs); + + compute_expr_stats(onerel, totalrows, + exprdata, nexprs, + rows, numrows); + + exprstats = serialize_expr_stats(exprdata, nexprs); + } } /* store the statistics in the catalog */ - statext_store(stat->statOid, ndistinct, dependencies, mcv, stats); + statext_store(stat->statOid, ndistinct, dependencies, mcv, exprstats, stats); /* for reporting progress */ pgstat_progress_update_param(PROGRESS_ANALYZE_EXT_STATS_COMPUTED, ++ext_cnt); + + pfree(exprs); } table_close(pg_stext, RowExclusiveLock); @@ -221,6 +269,10 @@ ComputeExtStatisticsRows(Relation onerel, MemoryContext oldcxt; int result = 0; + /* If there are no columns to analyze, just return 0. */ + if (!natts) + return 0; + cxt = AllocSetContextCreate(CurrentMemoryContext, "ComputeExtStatisticsRows", ALLOCSET_DEFAULT_SIZES); @@ -241,7 +293,7 @@ ComputeExtStatisticsRows(Relation onerel, * analyzed. If not, ignore it (don't report anything, we'll do that * during the actual build BuildRelationExtStatistics). */ - stats = lookup_var_attr_stats(onerel, stat->columns, + stats = lookup_var_attr_stats(onerel, stat->columns, stat->exprs, natts, vacattrstats); if (!stats) @@ -349,6 +401,10 @@ statext_is_kind_built(HeapTuple htup, char type) attnum = Anum_pg_statistic_ext_data_stxdmcv; break; + case STATS_EXT_EXPRESSIONS: + attnum = Anum_pg_statistic_ext_data_stxdexpr; + break; + default: elog(ERROR, "unexpected statistics type requested: %d", type); } @@ -388,6 +444,7 @@ fetch_statentries_for_relation(Relation pg_statext, Oid relid) ArrayType *arr; char *enabled; Form_pg_statistic_ext staForm; + List *exprs = NIL; entry = palloc0(sizeof(StatExtEntry)); staForm = (Form_pg_statistic_ext) GETSTRUCT(htup); @@ -415,10 +472,39 @@ fetch_statentries_for_relation(Relation pg_statext, Oid relid) { Assert((enabled[i] == STATS_EXT_NDISTINCT) || (enabled[i] == STATS_EXT_DEPENDENCIES) || - (enabled[i] == STATS_EXT_MCV)); + (enabled[i] == STATS_EXT_MCV) || + (enabled[i] == STATS_EXT_EXPRESSIONS)); entry->types = lappend_int(entry->types, (int) enabled[i]); } + /* decode expression (if any) */ + datum = SysCacheGetAttr(STATEXTOID, htup, + Anum_pg_statistic_ext_stxexprs, &isnull); + + if (!isnull) + { + char *exprsString; + + exprsString = TextDatumGetCString(datum); + exprs = (List *) stringToNode(exprsString); + + pfree(exprsString); + + /* + * Run the expressions through eval_const_expressions. This is not just an + * optimization, but is necessary, because the planner will be comparing + * them to similarly-processed qual clauses, and may fail to detect valid + * matches without this. We must not use canonicalize_qual, however, + * since these aren't qual expressions. + */ + exprs = (List *) eval_const_expressions(NULL, (Node *) exprs); + + /* May as well fix opfuncids too */ + fix_opfuncids((Node *) exprs); + } + + entry->exprs = exprs; + result = lappend(result, entry); } @@ -427,6 +513,86 @@ fetch_statentries_for_relation(Relation pg_statext, Oid relid) return result; } + +/* + * examine_attribute -- pre-analysis of a single column + * + * Determine whether the column is analyzable; if so, create and initialize + * a VacAttrStats struct for it. If not, return NULL. + */ +static VacAttrStats * +examine_attribute(Node *expr) +{ + HeapTuple typtuple; + VacAttrStats *stats; + int i; + bool ok; + + /* + * Create the VacAttrStats struct. Note that we only have a copy of the + * fixed fields of the pg_attribute tuple. + */ + stats = (VacAttrStats *) palloc0(sizeof(VacAttrStats)); + + /* fake the attribute */ + stats->attr = (Form_pg_attribute) palloc0(ATTRIBUTE_FIXED_PART_SIZE); + stats->attr->attstattarget = -1; + + /* + * When analyzing an expression index, believe the expression tree's type + * not the column datatype --- the latter might be the opckeytype storage + * type of the opclass, which is not interesting for our purposes. (Note: + * if we did anything with non-expression index columns, we'd need to + * figure out where to get the correct type info from, but for now that's + * not a problem.) It's not clear whether anyone will care about the + * typmod, but we store that too just in case. + */ + stats->attrtypid = exprType(expr); + stats->attrtypmod = exprTypmod(expr); + stats->attrcollid = exprCollation(expr); + + typtuple = SearchSysCacheCopy1(TYPEOID, + ObjectIdGetDatum(stats->attrtypid)); + if (!HeapTupleIsValid(typtuple)) + elog(ERROR, "cache lookup failed for type %u", stats->attrtypid); + stats->attrtype = (Form_pg_type) GETSTRUCT(typtuple); + // stats->anl_context = anl_context; /* FIXME? */ + stats->tupattnum = InvalidAttrNumber; + + /* + * The fields describing the stats->stavalues[n] element types default to + * the type of the data being analyzed, but the type-specific typanalyze + * function can change them if it wants to store something else. + */ + for (i = 0; i < STATISTIC_NUM_SLOTS; i++) + { + stats->statypid[i] = stats->attrtypid; + stats->statyplen[i] = stats->attrtype->typlen; + stats->statypbyval[i] = stats->attrtype->typbyval; + stats->statypalign[i] = stats->attrtype->typalign; + } + + /* + * Call the type-specific typanalyze function. If none is specified, use + * std_typanalyze(). + */ + if (OidIsValid(stats->attrtype->typanalyze)) + ok = DatumGetBool(OidFunctionCall1(stats->attrtype->typanalyze, + PointerGetDatum(stats))); + else + ok = std_typanalyze(stats); + + if (!ok || stats->compute_stats == NULL || stats->minrows <= 0) + { + heap_freetuple(typtuple); + pfree(stats->attr); + pfree(stats); + return NULL; + } + + return stats; +} + /* * Using 'vacatts' of size 'nvacatts' as input data, return a newly built * VacAttrStats array which includes only the items corresponding to @@ -435,15 +601,18 @@ fetch_statentries_for_relation(Relation pg_statext, Oid relid) * to the caller that the stats should not be built. */ static VacAttrStats ** -lookup_var_attr_stats(Relation rel, Bitmapset *attrs, +lookup_var_attr_stats(Relation rel, Bitmapset *attrs, List *exprs, int nvacatts, VacAttrStats **vacatts) { int i = 0; int x = -1; + int natts; VacAttrStats **stats; + ListCell *lc; + + natts = bms_num_members(attrs) + list_length(exprs); - stats = (VacAttrStats **) - palloc(bms_num_members(attrs) * sizeof(VacAttrStats *)); + stats = (VacAttrStats **) palloc(natts * sizeof(VacAttrStats *)); /* lookup VacAttrStats info for the requested columns (same attnum) */ while ((x = bms_next_member(attrs, x)) >= 0) @@ -480,6 +649,24 @@ lookup_var_attr_stats(Relation rel, Bitmapset *attrs, i++; } + /* also add info for expressions */ + foreach (lc, exprs) + { + Node *expr = (Node *) lfirst(lc); + + stats[i] = examine_attribute(expr); + + /* + * FIXME We need tuple descriptor later, and we just grab it from + * stats[0]->tupDesc (see e.g. statext_mcv_build). But as coded + * examine_attribute does not set that, so just grab it from the + * first vacatts element. + */ + stats[i]->tupDesc = vacatts[0]->tupDesc; + + i++; + } + return stats; } @@ -491,7 +678,7 @@ lookup_var_attr_stats(Relation rel, Bitmapset *attrs, static void statext_store(Oid statOid, MVNDistinct *ndistinct, MVDependencies *dependencies, - MCVList *mcv, VacAttrStats **stats) + MCVList *mcv, Datum exprs, VacAttrStats **stats) { Relation pg_stextdata; HeapTuple stup, @@ -532,11 +719,17 @@ statext_store(Oid statOid, nulls[Anum_pg_statistic_ext_data_stxdmcv - 1] = (data == NULL); values[Anum_pg_statistic_ext_data_stxdmcv - 1] = PointerGetDatum(data); } + if (exprs != (Datum) 0) + { + nulls[Anum_pg_statistic_ext_data_stxdexpr - 1] = false; + values[Anum_pg_statistic_ext_data_stxdexpr - 1] = exprs; + } /* always replace the value (either by bytea or NULL) */ replaces[Anum_pg_statistic_ext_data_stxdndistinct - 1] = true; replaces[Anum_pg_statistic_ext_data_stxddependencies - 1] = true; replaces[Anum_pg_statistic_ext_data_stxdmcv - 1] = true; + replaces[Anum_pg_statistic_ext_data_stxdexpr - 1] = true; /* there should already be a pg_statistic_ext_data tuple */ oldtup = SearchSysCache1(STATEXTDATASTXOID, ObjectIdGetDatum(statOid)); @@ -741,8 +934,9 @@ build_attnums_array(Bitmapset *attrs, int *numattrs) * can simply pfree the return value to release all of it. */ SortItem * -build_sorted_items(int numrows, int *nitems, HeapTuple *rows, TupleDesc tdesc, - MultiSortSupport mss, int numattrs, AttrNumber *attnums) +build_sorted_items(int numrows, int *nitems, HeapTuple *rows, ExprInfo *exprs, + TupleDesc tdesc, MultiSortSupport mss, + int numattrs, AttrNumber *attnums) { int i, j, @@ -789,8 +983,24 @@ build_sorted_items(int numrows, int *nitems, HeapTuple *rows, TupleDesc tdesc, { Datum value; bool isnull; + int attlen; - value = heap_getattr(rows[i], attnums[j], tdesc, &isnull); + if (attnums[j] <= MaxHeapAttributeNumber) + { + value = heap_getattr(rows[i], attnums[j], tdesc, &isnull); + attlen = TupleDescAttr(tdesc, attnums[j] - 1)->attlen; + } + else + { + int idx = EXPRESSION_INDEX(attnums[j]); + + Assert((idx >= 0) && (idx < exprs->nexprs)); + + value = exprs->values[idx][i]; + isnull = exprs->nulls[idx][i]; + + attlen = get_typlen(exprs->types[idx]); + } /* * If this is a varlena value, check if it's too wide and if yes @@ -801,8 +1011,7 @@ build_sorted_items(int numrows, int *nitems, HeapTuple *rows, TupleDesc tdesc, * on the assumption that those are small (below WIDTH_THRESHOLD) * and will be discarded at the end of analyze. */ - if ((!isnull) && - (TupleDescAttr(tdesc, attnums[j] - 1)->attlen == -1)) + if ((!isnull) && (attlen == -1)) { if (toast_raw_datum_size(value) > WIDTH_THRESHOLD) { @@ -861,6 +1070,63 @@ has_stats_of_kind(List *stats, char requiredkind) return false; } +/* + * stat_find_expression + * Search for an expression in statistics object's list of expressions. + * + * Returns the index of the expression in the statistics object's list of + * expressions, or -1 if not found. + */ +static int +stat_find_expression(StatisticExtInfo *stat, Node *expr) +{ + ListCell *lc; + int idx; + + idx = 0; + foreach(lc, stat->exprs) + { + Node *stat_expr = (Node *) lfirst(lc); + + if (equal(stat_expr, expr)) + return idx; + idx++; + } + + /* Expression not found */ + return -1; +} + +/* + * stat_covers_expressions + * Test whether a statistics object covers all expressions in a list. + * + * Returns true if all expressions are covered. If expr_idxs is non-NULL, it + * is populated with the indexes of the expressions found. + */ +static bool +stat_covers_expressions(StatisticExtInfo *stat, List *exprs, + Bitmapset **expr_idxs) +{ + ListCell *lc; + + foreach (lc, exprs) + { + Node *expr = (Node *) lfirst(lc); + int expr_idx; + + expr_idx = stat_find_expression(stat, expr); + if (expr_idx == -1) + return false; + + if (expr_idxs != NULL) + *expr_idxs = bms_add_member(*expr_idxs, expr_idx); + } + + /* If we reach here, all expressions are covered */ + return true; +} + /* * choose_best_statistics * Look for and return statistics with the specified 'requiredkind' which @@ -881,7 +1147,8 @@ has_stats_of_kind(List *stats, char requiredkind) */ StatisticExtInfo * choose_best_statistics(List *stats, char requiredkind, - Bitmapset **clause_attnums, int nclauses) + Bitmapset **clause_attnums, List **clause_exprs, + int nclauses) { ListCell *lc; StatisticExtInfo *best_match = NULL; @@ -892,7 +1159,8 @@ choose_best_statistics(List *stats, char requiredkind, { int i; StatisticExtInfo *info = (StatisticExtInfo *) lfirst(lc); - Bitmapset *matched = NULL; + Bitmapset *matched_attnums = NULL; + Bitmapset *matched_exprs = NULL; int num_matched; int numkeys; @@ -901,35 +1169,43 @@ choose_best_statistics(List *stats, char requiredkind, continue; /* - * Collect attributes in remaining (unestimated) clauses fully covered - * by this statistic object. + * Collect attributes and expressions in remaining (unestimated) + * clauses fully covered by this statistic object. */ for (i = 0; i < nclauses; i++) { + Bitmapset *expr_idxs = NULL; + /* ignore incompatible/estimated clauses */ - if (!clause_attnums[i]) + if (!clause_attnums[i] && !clause_exprs[i]) continue; /* ignore clauses that are not covered by this object */ - if (!bms_is_subset(clause_attnums[i], info->keys)) + if (!bms_is_subset(clause_attnums[i], info->keys) || + !stat_covers_expressions(info, clause_exprs[i], &expr_idxs)) continue; - matched = bms_add_members(matched, clause_attnums[i]); + /* record attnums and indexes of expressions covered */ + matched_attnums = bms_add_members(matched_attnums, clause_attnums[i]); + matched_exprs = bms_add_members(matched_exprs, expr_idxs); } - num_matched = bms_num_members(matched); - bms_free(matched); + num_matched = bms_num_members(matched_attnums) + bms_num_members(matched_exprs); + + bms_free(matched_attnums); + bms_free(matched_exprs); /* * save the actual number of keys in the stats so that we can choose * the narrowest stats with the most matching keys. */ - numkeys = bms_num_members(info->keys); + numkeys = bms_num_members(info->keys) + list_length(info->exprs); /* - * Use this object when it increases the number of matched clauses or - * when it matches the same number of attributes but these stats have - * fewer keys than any previous match. + * Use this object when it increases the number of matched attributes + * and expressions or when it matches the same number of attributes + * and expressions but these stats have fewer keys than any previous + * match. */ if (num_matched > best_num_matched || (num_matched == best_num_matched && numkeys < best_match_keys)) @@ -954,7 +1230,8 @@ choose_best_statistics(List *stats, char requiredkind, */ static bool statext_is_compatible_clause_internal(PlannerInfo *root, Node *clause, - Index relid, Bitmapset **attnums) + Index relid, Bitmapset **attnums, + List **exprs) { /* Look inside any binary-compatible relabeling (as in examine_variable) */ if (IsA(clause, RelabelType)) @@ -982,19 +1259,19 @@ statext_is_compatible_clause_internal(PlannerInfo *root, Node *clause, return true; } - /* (Var op Const) or (Const op Var) */ + /* (Var/Expr op Const) or (Const op Var/Expr) */ if (is_opclause(clause)) { RangeTblEntry *rte = root->simple_rte_array[relid]; OpExpr *expr = (OpExpr *) clause; - Var *var; + Node *clause_expr; /* Only expressions with two arguments are considered compatible. */ if (list_length(expr->args) != 2) return false; - /* Check if the expression has the right shape (one Var, one Const) */ - if (!examine_clause_args(expr->args, &var, NULL, NULL)) + /* Check if the expression has the right shape */ + if (!examine_opclause_args(expr->args, &clause_expr, NULL, NULL)) return false; /* @@ -1012,7 +1289,7 @@ statext_is_compatible_clause_internal(PlannerInfo *root, Node *clause, case F_SCALARLESEL: case F_SCALARGTSEL: case F_SCALARGESEL: - /* supported, will continue with inspection of the Var */ + /* supported, will continue with inspection of the Var/Expr */ break; default: @@ -1034,23 +1311,29 @@ statext_is_compatible_clause_internal(PlannerInfo *root, Node *clause, !get_func_leakproof(get_opcode(expr->opno))) return false; - return statext_is_compatible_clause_internal(root, (Node *) var, - relid, attnums); + /* Check (Var op Const) or (Const op Var) clauses by recursing. */ + if (IsA(clause_expr, Var)) + return statext_is_compatible_clause_internal(root, clause_expr, + relid, attnums, exprs); + + /* Otherwise we have (Expr op Const) or (Const op Expr). */ + *exprs = lappend(*exprs, clause_expr); + return true; } - /* Var IN Array */ + /* Var/Expr IN Array */ if (IsA(clause, ScalarArrayOpExpr)) { RangeTblEntry *rte = root->simple_rte_array[relid]; ScalarArrayOpExpr *expr = (ScalarArrayOpExpr *) clause; - Var *var; + Node *clause_expr; /* Only expressions with two arguments are considered compatible. */ if (list_length(expr->args) != 2) return false; /* Check if the expression has the right shape (one Var, one Const) */ - if (!examine_clause_args(expr->args, &var, NULL, NULL)) + if (!examine_opclause_args(expr->args, &clause_expr, NULL, NULL)) return false; /* @@ -1068,7 +1351,7 @@ statext_is_compatible_clause_internal(PlannerInfo *root, Node *clause, case F_SCALARLESEL: case F_SCALARGTSEL: case F_SCALARGESEL: - /* supported, will continue with inspection of the Var */ + /* supported, will continue with inspection of the Var/Expr */ break; default: @@ -1090,8 +1373,14 @@ statext_is_compatible_clause_internal(PlannerInfo *root, Node *clause, !get_func_leakproof(get_opcode(expr->opno))) return false; - return statext_is_compatible_clause_internal(root, (Node *) var, - relid, attnums); + /* Check Var IN Array clauses by recursing. */ + if (IsA(clause_expr, Var)) + return statext_is_compatible_clause_internal(root, clause_expr, + relid, attnums, exprs); + + /* Otherwise we have Expr IN Array. */ + *exprs = lappend(*exprs, clause_expr); + return true; } /* AND/OR/NOT clause */ @@ -1124,54 +1413,62 @@ statext_is_compatible_clause_internal(PlannerInfo *root, Node *clause, */ if (!statext_is_compatible_clause_internal(root, (Node *) lfirst(lc), - relid, attnums)) + relid, attnums, exprs)) return false; } return true; } - /* Var IS NULL */ + /* Var/Expr IS NULL */ if (IsA(clause, NullTest)) { NullTest *nt = (NullTest *) clause; - /* - * Only simple (Var IS NULL) expressions supported for now. Maybe we - * could use examine_variable to fix this? - */ - if (!IsA(nt->arg, Var)) - return false; + /* Check Var IS NULL clauses by recursing. */ + if (IsA(nt->arg, Var)) + return statext_is_compatible_clause_internal(root, (Node *) (nt->arg), + relid, attnums, exprs); - return statext_is_compatible_clause_internal(root, (Node *) (nt->arg), - relid, attnums); + /* Otherwise we have Expr IS NULL. */ + *exprs = lappend(*exprs, nt->arg); + return true; } - return false; + /* + * Treat any other expressions as bare expressions to be matched against + * expressions in statistics objects. + */ + *exprs = lappend(*exprs, clause); + return true; } /* * statext_is_compatible_clause * Determines if the clause is compatible with MCV lists. * - * Currently, we only support three types of clauses: + * Currently, we only support the following types of clauses: * - * (a) OpExprs of the form (Var op Const), or (Const op Var), where the op - * is one of ("=", "<", ">", ">=", "<=") + * (a) OpExprs of the form (Var/Expr op Const), or (Const op Var/Expr), where + * the op is one of ("=", "<", ">", ">=", "<=") * - * (b) (Var IS [NOT] NULL) + * (b) (Var/Expr IS [NOT] NULL) * * (c) combinations using AND/OR/NOT * + * (d) ScalarArrayOpExprs of the form (Var/Expr op ANY (array)) or (Var/Expr + * op ALL (array)) + * * In the future, the range of supported clauses may be expanded to more * complex cases, for example (Var op Var). */ static bool statext_is_compatible_clause(PlannerInfo *root, Node *clause, Index relid, - Bitmapset **attnums) + Bitmapset **attnums, List **exprs) { RangeTblEntry *rte = root->simple_rte_array[relid]; RestrictInfo *rinfo = (RestrictInfo *) clause; + int clause_relid; Oid userid; /* @@ -1191,7 +1488,7 @@ statext_is_compatible_clause(PlannerInfo *root, Node *clause, Index relid, foreach(lc, expr->args) { if (!statext_is_compatible_clause(root, (Node *) lfirst(lc), - relid, attnums)) + relid, attnums, exprs)) return false; } @@ -1206,25 +1503,37 @@ statext_is_compatible_clause(PlannerInfo *root, Node *clause, Index relid, if (rinfo->pseudoconstant) return false; - /* clauses referencing multiple varnos are incompatible */ - if (bms_membership(rinfo->clause_relids) != BMS_SINGLETON) + /* Clauses referencing other varnos are incompatible. */ + if (!bms_get_singleton_member(rinfo->clause_relids, &clause_relid) || + clause_relid != relid) return false; /* Check the clause and determine what attributes it references. */ if (!statext_is_compatible_clause_internal(root, (Node *) rinfo->clause, - relid, attnums)) + relid, attnums, exprs)) return false; /* - * Check that the user has permission to read all these attributes. Use - * checkAsUser if it's set, in case we're accessing the table via a view. + * Check that the user has permission to read all required attributes. + * Use checkAsUser if it's set, in case we're accessing the table via a + * view. */ userid = rte->checkAsUser ? rte->checkAsUser : GetUserId(); if (pg_class_aclcheck(rte->relid, userid, ACL_SELECT) != ACLCHECK_OK) { + Bitmapset *clause_attnums; + /* Don't have table privilege, must check individual columns */ - if (bms_is_member(InvalidAttrNumber, *attnums)) + if (*exprs != NIL) + { + pull_varattnos((Node *) exprs, relid, &clause_attnums); + clause_attnums = bms_add_members(clause_attnums, *attnums); + } + else + clause_attnums = *attnums; + + if (bms_is_member(InvalidAttrNumber, clause_attnums)) { /* Have a whole-row reference, must have access to all columns */ if (pg_attribute_aclcheck_all(rte->relid, userid, ACL_SELECT, @@ -1236,7 +1545,7 @@ statext_is_compatible_clause(PlannerInfo *root, Node *clause, Index relid, /* Check the columns referenced by the clause */ int attnum = -1; - while ((attnum = bms_next_member(*attnums, attnum)) >= 0) + while ((attnum = bms_next_member(clause_attnums, attnum)) >= 0) { if (pg_attribute_aclcheck(rte->relid, attnum, userid, ACL_SELECT) != ACLCHECK_OK) @@ -1290,7 +1599,8 @@ statext_mcv_clauselist_selectivity(PlannerInfo *root, List *clauses, int varReli bool is_or) { ListCell *l; - Bitmapset **list_attnums; + Bitmapset **list_attnums; /* attnums extracted from the clause */ + List **list_exprs; /* expressions matched to any statistic */ int listidx; Selectivity sel = (is_or) ? 0.0 : 1.0; @@ -1301,13 +1611,16 @@ statext_mcv_clauselist_selectivity(PlannerInfo *root, List *clauses, int varReli list_attnums = (Bitmapset **) palloc(sizeof(Bitmapset *) * list_length(clauses)); + /* expressions extracted from complex expressions */ + list_exprs = (List **) palloc(sizeof(Node *) * list_length(clauses)); + /* - * Pre-process the clauses list to extract the attnums seen in each item. - * We need to determine if there's any clauses which will be useful for - * selectivity estimations with extended stats. Along the way we'll record - * all of the attnums for each clause in a list which we'll reference - * later so we don't need to repeat the same work again. We'll also keep - * track of all attnums seen. + * Pre-process the clauses list to extract the attnums and expressions + * seen in each item. We need to determine if there are any clauses which + * will be useful for selectivity estimations with extended stats. Along + * the way we'll record all of the attnums and expressions for each clause + * in lists which we'll reference later so we don't need to repeat the + * same work again. * * We also skip clauses that we already estimated using different types of * statistics (we treat them as incompatible). @@ -1317,12 +1630,19 @@ statext_mcv_clauselist_selectivity(PlannerInfo *root, List *clauses, int varReli { Node *clause = (Node *) lfirst(l); Bitmapset *attnums = NULL; + List *exprs = NIL; if (!bms_is_member(listidx, *estimatedclauses) && - statext_is_compatible_clause(root, clause, rel->relid, &attnums)) + statext_is_compatible_clause(root, clause, rel->relid, &attnums, &exprs)) + { list_attnums[listidx] = attnums; + list_exprs[listidx] = exprs; + } else + { list_attnums[listidx] = NULL; + list_exprs[listidx] = NIL; + } listidx++; } @@ -1336,7 +1656,8 @@ statext_mcv_clauselist_selectivity(PlannerInfo *root, List *clauses, int varReli /* find the best suited statistics object for these attnums */ stat = choose_best_statistics(rel->statlist, STATS_EXT_MCV, - list_attnums, list_length(clauses)); + list_attnums, list_exprs, + list_length(clauses)); /* * if no (additional) matching stats could be found then we've nothing @@ -1351,28 +1672,39 @@ statext_mcv_clauselist_selectivity(PlannerInfo *root, List *clauses, int varReli /* now filter the clauses to be estimated using the selected MCV */ stat_clauses = NIL; - /* record which clauses are simple (single column) */ + /* record which clauses are simple (single column or expression) */ simple_clauses = NULL; listidx = 0; foreach(l, clauses) { /* - * If the clause is compatible with the selected statistics, mark - * it as estimated and add it to the list to estimate. + * If the clause is not already estimated and is compatible with + * the selected statistics object (all attributes and expressions + * covered), mark it as estimated and add it to the list to + * estimate. */ - if (list_attnums[listidx] != NULL && - bms_is_subset(list_attnums[listidx], stat->keys)) + if (!bms_is_member(listidx, *estimatedclauses) && + bms_is_subset(list_attnums[listidx], stat->keys) && + stat_covers_expressions(stat, list_exprs[listidx], NULL)) { - if (bms_membership(list_attnums[listidx]) == BMS_SINGLETON) + /* record simple clauses (single column or expression) */ + if ((list_attnums[listidx] == NULL && + list_length(list_exprs[listidx]) == 1) || + (list_exprs[listidx] == NIL && + bms_membership(list_attnums[listidx]) == BMS_SINGLETON)) simple_clauses = bms_add_member(simple_clauses, list_length(stat_clauses)); + /* add clause to list and mark as estimated */ stat_clauses = lappend(stat_clauses, (Node *) lfirst(l)); *estimatedclauses = bms_add_member(*estimatedclauses, listidx); bms_free(list_attnums[listidx]); list_attnums[listidx] = NULL; + + list_free(list_exprs[listidx]); + list_exprs[listidx] = NULL; } listidx++; @@ -1561,23 +1893,24 @@ statext_clauselist_selectivity(PlannerInfo *root, List *clauses, int varRelid, } /* - * examine_opclause_expression - * Split expression into Var and Const parts. + * examine_opclause_args + * Split an operator expression's arguments into Expr and Const parts. * - * Attempts to match the arguments to either (Var op Const) or (Const op Var), - * possibly with a RelabelType on top. When the expression matches this form, - * returns true, otherwise returns false. + * Attempts to match the arguments to either (Expr op Const) or (Const op + * Expr), possibly with a RelabelType on top. When the expression matches this + * form, returns true, otherwise returns false. * - * Optionally returns pointers to the extracted Var/Const nodes, when passed - * non-null pointers (varp, cstp and varonleftp). The varonleftp flag specifies - * on which side of the operator we found the Var node. + * Optionally returns pointers to the extracted Expr/Const nodes, when passed + * non-null pointers (exprp, cstp and expronleftp). The expronleftp flag + * specifies on which side of the operator we found the expression node. */ bool -examine_clause_args(List *args, Var **varp, Const **cstp, bool *varonleftp) +examine_opclause_args(List *args, Node **exprp, Const **cstp, + bool *expronleftp) { - Var *var; + Node *expr; Const *cst; - bool varonleft; + bool expronleft; Node *leftop, *rightop; @@ -1594,30 +1927,665 @@ examine_clause_args(List *args, Var **varp, Const **cstp, bool *varonleftp) if (IsA(rightop, RelabelType)) rightop = (Node *) ((RelabelType *) rightop)->arg; - if (IsA(leftop, Var) && IsA(rightop, Const)) + if (IsA(rightop, Const)) { - var = (Var *) leftop; + expr = (Node *) leftop; cst = (Const *) rightop; - varonleft = true; + expronleft = true; } - else if (IsA(leftop, Const) && IsA(rightop, Var)) + else if (IsA(leftop, Const)) { - var = (Var *) rightop; + expr = (Node *) rightop; cst = (Const *) leftop; - varonleft = false; + expronleft = false; } else return false; /* return pointers to the extracted parts if requested */ - if (varp) - *varp = var; + if (exprp) + *exprp = expr; if (cstp) *cstp = cst; - if (varonleftp) - *varonleftp = varonleft; + if (expronleftp) + *expronleftp = expronleft; return true; } + + +/* + * Compute statistics about expressions of a relation. + */ +static void +compute_expr_stats(Relation onerel, double totalrows, + AnlExprData *exprdata, int nexprs, + HeapTuple *rows, int numrows) +{ + MemoryContext expr_context, + old_context; + int ind, + i; + + expr_context = AllocSetContextCreate(CurrentMemoryContext, + "Analyze Expression", + ALLOCSET_DEFAULT_SIZES); + old_context = MemoryContextSwitchTo(expr_context); + + for (ind = 0; ind < nexprs; ind++) + { + AnlExprData *thisdata = &exprdata[ind]; + VacAttrStats *stats = thisdata->vacattrstat; + Node *expr = thisdata->expr; + TupleTableSlot *slot; + EState *estate; + ExprContext *econtext; + Datum *exprvals; + bool *exprnulls; + ExprState *exprstate; + int tcnt; + + /* Are we still in the main context? */ + Assert(CurrentMemoryContext == expr_context); + + /* + * Need an EState for evaluation of expressions. Create it in + * the per-expression context to be sure it gets cleaned up at + * the bottom of the loop. + */ + estate = CreateExecutorState(); + econtext = GetPerTupleExprContext(estate); + + /* Set up expression evaluation state */ + exprstate = ExecPrepareExpr((Expr *) expr, estate); + + /* Need a slot to hold the current heap tuple, too */ + slot = MakeSingleTupleTableSlot(RelationGetDescr(onerel), + &TTSOpsHeapTuple); + + /* Arrange for econtext's scan tuple to be the tuple under test */ + econtext->ecxt_scantuple = slot; + + /* Compute and save index expression values */ + exprvals = (Datum *) palloc(numrows * sizeof(Datum)); + exprnulls = (bool *) palloc(numrows * sizeof(bool)); + + tcnt = 0; + for (i = 0; i < numrows; i++) + { + Datum datum; + bool isnull; + + /* + * Reset the per-tuple context each time, to reclaim any cruft + * left behind by evaluating the predicate or index expressions. + */ + ResetExprContext(econtext); + + /* Set up for expression evaluation */ + ExecStoreHeapTuple(rows[i], slot, false); + + /* + * Evaluate the expression. We do this in the per-tuple context + * so as not to leak memory, and then copy the result into the + * context created at the beginning of this function. + */ + datum = ExecEvalExprSwitchContext(exprstate, + GetPerTupleExprContext(estate), + &isnull); + if (isnull) + { + exprvals[tcnt] = (Datum) 0; + exprnulls[tcnt] = true; + } + else + { + /* Make sure we copy the data into the context. */ + Assert(CurrentMemoryContext == expr_context); + + exprvals[tcnt] = datumCopy(datum, + stats->attrtype->typbyval, + stats->attrtype->typlen); + exprnulls[tcnt] = false; + } + + tcnt++; + } + + /* + * Now we can compute the statistics for the expression columns. + * + * XXX Unlike compute_index_stats we don't need to switch and reset + * memory contexts here, because we're only computing stats for a + * single expression (and not iterating over many indexes), so we + * just do it in expr_context. Note that compute_stats copies the + * result into stats->anl_context, so it does not disappear. + */ + if (tcnt > 0) + { + AttributeOpts *aopt = + get_attribute_options(stats->attr->attrelid, + stats->attr->attnum); + + stats->exprvals = exprvals; + stats->exprnulls = exprnulls; + stats->rowstride = 1; + stats->compute_stats(stats, + expr_fetch_func, + tcnt, + tcnt); + + /* + * If the n_distinct option is specified, it overrides the + * above computation. + */ + if (aopt != NULL && aopt->n_distinct != 0.0) + stats->stadistinct = aopt->n_distinct; + } + + /* And clean up */ + MemoryContextSwitchTo(expr_context); + + ExecDropSingleTupleTableSlot(slot); + FreeExecutorState(estate); + MemoryContextResetAndDeleteChildren(expr_context); + } + + MemoryContextSwitchTo(old_context); + MemoryContextDelete(expr_context); +} + + +/* + * Fetch function for analyzing index expressions. + * + * We have not bothered to construct index tuples, instead the data is + * just in Datum arrays. + */ +static Datum +expr_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull) +{ + int i; + + /* exprvals and exprnulls are already offset for proper column */ + i = rownum * stats->rowstride; + *isNull = stats->exprnulls[i]; + return stats->exprvals[i]; +} + +/* + * Build analyze data for a list of expressions. As this is not tied + * directly to a relation (table or index), we have to fake some of + * the data. + */ +static AnlExprData * +build_expr_data(List *exprs) +{ + int idx; + int nexprs = list_length(exprs); + AnlExprData *exprdata; + ListCell *lc; + + exprdata = (AnlExprData *) palloc0(nexprs * sizeof(AnlExprData)); + + idx = 0; + foreach (lc, exprs) + { + Node *expr = (Node *) lfirst(lc); + AnlExprData *thisdata = &exprdata[idx]; + + thisdata->expr = expr; + thisdata->vacattrstat = (VacAttrStats *) palloc(sizeof(VacAttrStats)); + + thisdata->vacattrstat = examine_expression(expr); + idx++; + } + + return exprdata; +} + +/* + * examine_expression -- pre-analysis of a single column + * + * Determine whether the column is analyzable; if so, create and initialize + * a VacAttrStats struct for it. If not, return NULL. + */ +static VacAttrStats * +examine_expression(Node *expr) +{ + HeapTuple typtuple; + VacAttrStats *stats; + int i; + bool ok; + + Assert(expr != NULL); + + /* + * Create the VacAttrStats struct. + */ + stats = (VacAttrStats *) palloc0(sizeof(VacAttrStats)); + + /* + * When analyzing an expression, believe the expression tree's type. + */ + stats->attrtypid = exprType(expr); + stats->attrtypmod = exprTypmod(expr); + + /* + * XXX Do we need to do anything special about the collation, similar + * to what examine_attribute does for expression indexes? + */ + stats->attrcollid = exprCollation(expr); + + /* + * We don't have any pg_attribute for expressions, so let's fake + * something reasonable into attstattarget, which is the only thing + * std_typanalyze needs. + */ + stats->attr = (Form_pg_attribute) palloc(ATTRIBUTE_FIXED_PART_SIZE); + + /* + * FIXME we should probably get the target from the extended stats + * object, or something like that. + */ + stats->attr->attstattarget = default_statistics_target; + + /* initialize some basic fields */ + stats->attr->attrelid = InvalidOid; + stats->attr->attnum = InvalidAttrNumber; + stats->attr->atttypid = stats->attrtypid; + + typtuple = SearchSysCacheCopy1(TYPEOID, + ObjectIdGetDatum(stats->attrtypid)); + if (!HeapTupleIsValid(typtuple)) + elog(ERROR, "cache lookup failed for type %u", stats->attrtypid); + stats->attrtype = (Form_pg_type) GETSTRUCT(typtuple); + stats->anl_context = CurrentMemoryContext; /* XXX should be using something else? */ + stats->tupattnum = InvalidAttrNumber; + + /* + * The fields describing the stats->stavalues[n] element types default to + * the type of the data being analyzed, but the type-specific typanalyze + * function can change them if it wants to store something else. + */ + for (i = 0; i < STATISTIC_NUM_SLOTS; i++) + { + stats->statypid[i] = stats->attrtypid; + stats->statyplen[i] = stats->attrtype->typlen; + stats->statypbyval[i] = stats->attrtype->typbyval; + stats->statypalign[i] = stats->attrtype->typalign; + } + + /* + * Call the type-specific typanalyze function. If none is specified, use + * std_typanalyze(). + */ + if (OidIsValid(stats->attrtype->typanalyze)) + ok = DatumGetBool(OidFunctionCall1(stats->attrtype->typanalyze, + PointerGetDatum(stats))); + else + ok = std_typanalyze(stats); + + if (!ok || stats->compute_stats == NULL || stats->minrows <= 0) + { + heap_freetuple(typtuple); + pfree(stats); + return NULL; + } + + return stats; +} + +/* form an array of pg_statistic rows (per update_attstats) */ +static Datum +serialize_expr_stats(AnlExprData *exprdata, int nexprs) +{ + int exprno; + Oid typOid; + Relation sd; + + ArrayBuildState *astate = NULL; + + sd = table_open(StatisticRelationId, RowExclusiveLock); + + /* lookup OID of composite type for pg_statistic */ + typOid = get_rel_type_id(StatisticRelationId); + if (!OidIsValid(typOid)) + ereport(ERROR, + (errcode(ERRCODE_WRONG_OBJECT_TYPE), + errmsg("relation \"pg_statistic\" does not have a composite type"))); + + for (exprno = 0; exprno < nexprs; exprno++) + { + int i, k; + VacAttrStats *stats = exprdata[exprno].vacattrstat; + + Datum values[Natts_pg_statistic]; + bool nulls[Natts_pg_statistic]; + HeapTuple stup; + + if (!stats->stats_valid) + { + astate = accumArrayResult(astate, + (Datum) 0, + true, + typOid, + CurrentMemoryContext); + continue; + } + + /* + * Construct a new pg_statistic tuple + */ + for (i = 0; i < Natts_pg_statistic; ++i) + { + nulls[i] = false; + } + + values[Anum_pg_statistic_starelid - 1] = ObjectIdGetDatum(InvalidOid); + values[Anum_pg_statistic_staattnum - 1] = Int16GetDatum(InvalidAttrNumber); + values[Anum_pg_statistic_stainherit - 1] = BoolGetDatum(false); + values[Anum_pg_statistic_stanullfrac - 1] = Float4GetDatum(stats->stanullfrac); + values[Anum_pg_statistic_stawidth - 1] = Int32GetDatum(stats->stawidth); + values[Anum_pg_statistic_stadistinct - 1] = Float4GetDatum(stats->stadistinct); + i = Anum_pg_statistic_stakind1 - 1; + for (k = 0; k < STATISTIC_NUM_SLOTS; k++) + { + values[i++] = Int16GetDatum(stats->stakind[k]); /* stakindN */ + } + i = Anum_pg_statistic_staop1 - 1; + for (k = 0; k < STATISTIC_NUM_SLOTS; k++) + { + values[i++] = ObjectIdGetDatum(stats->staop[k]); /* staopN */ + } + i = Anum_pg_statistic_stacoll1 - 1; + for (k = 0; k < STATISTIC_NUM_SLOTS; k++) + { + values[i++] = ObjectIdGetDatum(stats->stacoll[k]); /* stacollN */ + } + i = Anum_pg_statistic_stanumbers1 - 1; + for (k = 0; k < STATISTIC_NUM_SLOTS; k++) + { + int nnum = stats->numnumbers[k]; + + if (nnum > 0) + { + int n; + Datum *numdatums = (Datum *) palloc(nnum * sizeof(Datum)); + ArrayType *arry; + + for (n = 0; n < nnum; n++) + numdatums[n] = Float4GetDatum(stats->stanumbers[k][n]); + /* XXX knows more than it should about type float4: */ + arry = construct_array(numdatums, nnum, + FLOAT4OID, + sizeof(float4), true, TYPALIGN_INT); + values[i++] = PointerGetDatum(arry); /* stanumbersN */ + } + else + { + nulls[i] = true; + values[i++] = (Datum) 0; + } + } + i = Anum_pg_statistic_stavalues1 - 1; + for (k = 0; k < STATISTIC_NUM_SLOTS; k++) + { + if (stats->numvalues[k] > 0) + { + ArrayType *arry; + + arry = construct_array(stats->stavalues[k], + stats->numvalues[k], + stats->statypid[k], + stats->statyplen[k], + stats->statypbyval[k], + stats->statypalign[k]); + values[i++] = PointerGetDatum(arry); /* stavaluesN */ + } + else + { + nulls[i] = true; + values[i++] = (Datum) 0; + } + } + + stup = heap_form_tuple(RelationGetDescr(sd), values, nulls); + + astate = accumArrayResult(astate, + heap_copy_tuple_as_datum(stup, RelationGetDescr(sd)), + false, + typOid, + CurrentMemoryContext); + } + + table_close(sd, RowExclusiveLock); + + return makeArrayResult(astate, CurrentMemoryContext); +} + + +/* + * Loads pg_statistic record from expression statistics for expression + * identified by the supplied index. + */ +HeapTuple +statext_expressions_load(Oid stxoid, int idx) +{ + bool isnull; + Datum value; + HeapTuple htup; + ExpandedArrayHeader *eah; + HeapTupleHeader td; + HeapTupleData tmptup; + HeapTuple tup; + + htup = SearchSysCache1(STATEXTDATASTXOID, ObjectIdGetDatum(stxoid)); + if (!HeapTupleIsValid(htup)) + elog(ERROR, "cache lookup failed for statistics object %u", stxoid); + + value = SysCacheGetAttr(STATEXTDATASTXOID, htup, + Anum_pg_statistic_ext_data_stxdexpr, &isnull); + if (isnull) + elog(ERROR, + "requested statistics kind \"%c\" is not yet built for statistics object %u", + STATS_EXT_DEPENDENCIES, stxoid); + + eah = DatumGetExpandedArray(value); + + deconstruct_expanded_array(eah); + + td = DatumGetHeapTupleHeader(eah->dvalues[idx]); + + /* Build a temporary HeapTuple control structure */ + tmptup.t_len = HeapTupleHeaderGetDatumLength(td); + tmptup.t_data = td; + + tup = heap_copytuple(&tmptup); + + ReleaseSysCache(htup); + + return tup; +} + +/* + * Evaluate the expressions, so that we can use the results to build + * all the requested statistics types. This matters especially for + * expensive expressions, of course. + */ +static ExprInfo * +evaluate_expressions(Relation rel, List *exprs, int numrows, HeapTuple *rows) +{ + /* evaluated expressions */ + ExprInfo *result; + char *ptr; + Size len; + + int i; + int idx; + TupleTableSlot *slot; + EState *estate; + ExprContext *econtext; + List *exprstates = NIL; + int nexprs = list_length(exprs); + ListCell *lc; + + /* allocate everything as a single chunk, so we can free it easily */ + len = MAXALIGN(sizeof(ExprInfo)); + len += MAXALIGN(sizeof(Oid) * nexprs); /* types */ + len += MAXALIGN(sizeof(Oid) * nexprs); /* collations */ + + /* values */ + len += MAXALIGN(sizeof(Datum *) * nexprs); + len += nexprs * MAXALIGN(sizeof(Datum) * numrows); + + /* nulls */ + len += MAXALIGN(sizeof(bool *) * nexprs); + len += nexprs * MAXALIGN(sizeof(bool) * numrows); + + ptr = palloc(len); + + /* set the pointers */ + result = (ExprInfo *) ptr; + ptr += MAXALIGN(sizeof(ExprInfo)); + + /* types */ + result->types = (Oid *) ptr; + ptr += MAXALIGN(sizeof(Oid) * nexprs); + + /* collations */ + result->collations = (Oid *) ptr; + ptr += MAXALIGN(sizeof(Oid) * nexprs); + + /* values */ + result->values = (Datum **) ptr; + ptr += MAXALIGN(sizeof(Datum *) * nexprs); + + /* nulls */ + result->nulls = (bool **) ptr; + ptr += MAXALIGN(sizeof(bool *) * nexprs); + + for (i = 0; i < nexprs; i++) + { + result->values[i] = (Datum *) ptr; + ptr += MAXALIGN(sizeof(Datum) * numrows); + + result->nulls[i] = (bool *) ptr; + ptr += MAXALIGN(sizeof(bool) * numrows); + } + + Assert((ptr - (char *) result) == len); + + result->nexprs = list_length(exprs); + + idx = 0; + foreach (lc, exprs) + { + Node *expr = (Node *) lfirst(lc); + + result->types[idx] = exprType(expr); + result->collations[idx] = exprCollation(expr); + + idx++; + } + + /* + * Need an EState for evaluation of index expressions and + * partial-index predicates. Create it in the per-index context to be + * sure it gets cleaned up at the bottom of the loop. + */ + estate = CreateExecutorState(); + econtext = GetPerTupleExprContext(estate); + + /* Need a slot to hold the current heap tuple, too */ + slot = MakeSingleTupleTableSlot(RelationGetDescr(rel), + &TTSOpsHeapTuple); + + /* Arrange for econtext's scan tuple to be the tuple under test */ + econtext->ecxt_scantuple = slot; + + /* Set up expression evaluation state */ + exprstates = ExecPrepareExprList(exprs, estate); + + for (i = 0; i < numrows; i++) + { + /* + * Reset the per-tuple context each time, to reclaim any cruft + * left behind by evaluating the predicate or index expressions. + */ + ResetExprContext(econtext); + + /* Set up for predicate or expression evaluation */ + ExecStoreHeapTuple(rows[i], slot, false); + + idx = 0; + foreach (lc, exprstates) + { + Datum datum; + bool isnull; + ExprState *exprstate = (ExprState *) lfirst(lc); + + /* + * FIXME this probably leaks memory. Maybe we should use + * ExecEvalExprSwitchContext but then we need to copy the + * result somewhere else. + */ + datum = ExecEvalExpr(exprstate, + GetPerTupleExprContext(estate), + &isnull); + if (isnull) + { + result->values[idx][i] = (Datum) 0; + result->nulls[idx][i] = true; + } + else + { + result->values[idx][i] = (Datum) datum; + result->nulls[idx][i] = false; + } + + idx++; + } + } + + ExecDropSingleTupleTableSlot(slot); + FreeExecutorState(estate); + + return result; +} + +/* + * add_expressions_to_attributes + * add expressions as attributes with high attnums + * + * Treat the expressions as attributes with attnums above the regular + * attnum range. This will allow us to handle everything in the same + * way, and identify expressions in the dependencies. + * + * XXX This always creates a copy of the bitmap. We might optimize this + * by only creating the copy with (nexprs > 0) but then we'd have to track + * this in order to free it (if we want to). Does not seem worth it. + */ +Bitmapset * +add_expressions_to_attributes(Bitmapset *attrs, int nexprs) +{ + int i; + + /* + * Copy the bitmapset and add fake attnums representing expressions, + * starting above MaxHeapAttributeNumber. + */ + attrs = bms_copy(attrs); + + /* start with (MaxHeapAttributeNumber + 1) */ + for (i = 0; i < nexprs; i++) + { + Assert(EXPRESSION_ATTNUM(i) > MaxHeapAttributeNumber); + + attrs = bms_add_member(attrs, EXPRESSION_ATTNUM(i)); + } + + return attrs; +} diff --git a/src/backend/statistics/mcv.c b/src/backend/statistics/mcv.c index abbc1f1ba8..3bb6fa733d 100644 --- a/src/backend/statistics/mcv.c +++ b/src/backend/statistics/mcv.c @@ -74,7 +74,8 @@ ((ndims) * sizeof(DimensionInfo)) + \ ((nitems) * ITEM_SIZE(ndims))) -static MultiSortSupport build_mss(VacAttrStats **stats, int numattrs); +static MultiSortSupport build_mss(VacAttrStats **stats, int numattrs, + ExprInfo *exprs); static SortItem *build_distinct_groups(int numrows, SortItem *items, MultiSortSupport mss, int *ndistinct); @@ -181,8 +182,9 @@ get_mincount_for_mcv_list(int samplerows, double totalrows) * */ MCVList * -statext_mcv_build(int numrows, HeapTuple *rows, Bitmapset *attrs, - VacAttrStats **stats, double totalrows, int stattarget) +statext_mcv_build(int numrows, HeapTuple *rows, ExprInfo *exprs, + Bitmapset *attrs, VacAttrStats **stats, + double totalrows, int stattarget) { int i, numattrs, @@ -195,14 +197,23 @@ statext_mcv_build(int numrows, HeapTuple *rows, Bitmapset *attrs, MCVList *mcvlist = NULL; MultiSortSupport mss; - attnums = build_attnums_array(attrs, &numattrs); - /* comparator for all the columns */ - mss = build_mss(stats, numattrs); + mss = build_mss(stats, bms_num_members(attrs), exprs); + + /* + * treat expressions as special attributes with high attnums + * + * XXX We do this after build_mss, because that expects the bitmapset + * to only contain simple attributes (with a matching VacAttrStats) + */ + attrs = add_expressions_to_attributes(attrs, exprs->nexprs); + + /* now build the array, with the special expression attnums */ + attnums = build_attnums_array(attrs, &numattrs); /* sort the rows */ - items = build_sorted_items(numrows, &nitems, rows, stats[0]->tupDesc, - mss, numattrs, attnums); + items = build_sorted_items(numrows, &nitems, rows, exprs, + stats[0]->tupDesc, mss, numattrs, attnums); if (!items) return NULL; @@ -338,6 +349,7 @@ statext_mcv_build(int numrows, HeapTuple *rows, Bitmapset *attrs, pfree(items); pfree(groups); + pfree(attrs); return mcvlist; } @@ -347,12 +359,12 @@ statext_mcv_build(int numrows, HeapTuple *rows, Bitmapset *attrs, * build MultiSortSupport for the attributes passed in attrs */ static MultiSortSupport -build_mss(VacAttrStats **stats, int numattrs) +build_mss(VacAttrStats **stats, int numattrs, ExprInfo *exprs) { int i; /* Sort by multiple columns (using array of SortSupport) */ - MultiSortSupport mss = multi_sort_init(numattrs); + MultiSortSupport mss = multi_sort_init(numattrs + exprs->nexprs); /* prepare the sort functions for all the attributes */ for (i = 0; i < numattrs; i++) @@ -368,6 +380,20 @@ build_mss(VacAttrStats **stats, int numattrs) multi_sort_add_dimension(mss, i, type->lt_opr, colstat->attrcollid); } + /* prepare the sort functions for all the expressions */ + for (i = 0; i < exprs->nexprs; i++) + { + TypeCacheEntry *type; + + type = lookup_type_cache(exprs->types[i], TYPECACHE_LT_OPR); + if (type->lt_opr == InvalidOid) /* shouldn't happen */ + elog(ERROR, "cache lookup failed for ordering operator for type %u", + exprs->types[i]); + + multi_sort_add_dimension(mss, numattrs + i, type->lt_opr, + exprs->collations[i]); + } + return mss; } @@ -570,7 +596,7 @@ statext_mcv_load(Oid mvoid) if (isnull) elog(ERROR, - "requested statistic kind \"%c\" is not yet built for statistics object %u", + "requested statistics kind \"%c\" is not yet built for statistics object %u", STATS_EXT_DEPENDENCIES, mvoid); result = statext_mcv_deserialize(DatumGetByteaP(mcvlist)); @@ -1523,6 +1549,59 @@ pg_mcv_list_send(PG_FUNCTION_ARGS) return byteasend(fcinfo); } +/* + * match the attribute to a dimension of the statistic + * + * Match the attribute/expression to statistics dimension. Optionally + * determine the collation. + */ +static int +mcv_match_expression(Node *expr, Bitmapset *keys, List *exprs, Oid *collid) +{ + int idx = -1; + + if (IsA(expr, Var)) + { + /* simple Var, so just lookup using varattno */ + Var *var = (Var *) expr; + + if (collid) + *collid = var->varcollid; + + idx = bms_member_index(keys, var->varattno); + + /* make sure the index is valid */ + Assert((idx >= 0) && (idx <= bms_num_members(keys))); + } + else + { + ListCell *lc; + + /* expressions are stored after the simple columns */ + idx = bms_num_members(keys); + + if (collid) + *collid = exprCollation(expr); + + /* expression - lookup in stats expressions */ + foreach(lc, exprs) + { + Node *stat_expr = (Node *) lfirst(lc); + + if (equal(expr, stat_expr)) + break; + + idx++; + } + + /* make sure the index is valid */ + Assert((idx >= bms_num_members(keys)) && + (idx <= bms_num_members(keys) + list_length(exprs))); + } + + return idx; +} + /* * mcv_get_match_bitmap * Evaluate clauses using the MCV list, and update the match bitmap. @@ -1541,10 +1620,14 @@ pg_mcv_list_send(PG_FUNCTION_ARGS) * the size to ~1/8. It would also allow us to combine bitmaps simply using * & and |, which should be faster than min/max. The bitmaps are fairly * small, though (thanks to the cap on the MCV list size). + * + * XXX There's a lot of code duplication between branches for simple columns + * and complex expressions. We should refactor it somehow. */ static bool * mcv_get_match_bitmap(PlannerInfo *root, List *clauses, - Bitmapset *keys, MCVList *mcvlist, bool is_or) + Bitmapset *keys, List *exprs, + MCVList *mcvlist, bool is_or) { int i; ListCell *l; @@ -1582,77 +1665,77 @@ mcv_get_match_bitmap(PlannerInfo *root, List *clauses, OpExpr *expr = (OpExpr *) clause; FmgrInfo opproc; - /* valid only after examine_clause_args returns true */ - Var *var; + /* valid only after examine_opclause_args returns true */ + Node *clause_expr; Const *cst; - bool varonleft; + bool expronleft; + int idx; + Oid collid; fmgr_info(get_opcode(expr->opno), &opproc); - /* extract the var and const from the expression */ - if (examine_clause_args(expr->args, &var, &cst, &varonleft)) - { - int idx; + /* extract the var/expr and const from the expression */ + if (!examine_opclause_args(expr->args, &clause_expr, &cst, &expronleft)) + elog(ERROR, "incompatible clause"); - /* match the attribute to a dimension of the statistic */ - idx = bms_member_index(keys, var->varattno); + /* match the attribute/expression to a dimension of the statistic */ + idx = mcv_match_expression(clause_expr, keys, exprs, &collid); + + /* + * Walk through the MCV items and evaluate the current clause. + * We can skip items that were already ruled out, and + * terminate if there are no remaining MCV items that might + * possibly match. + */ + for (i = 0; i < mcvlist->nitems; i++) + { + bool match = true; + MCVItem *item = &mcvlist->items[i]; /* - * Walk through the MCV items and evaluate the current clause. - * We can skip items that were already ruled out, and - * terminate if there are no remaining MCV items that might - * possibly match. + * When the MCV item or the Const value is NULL we can + * treat this as a mismatch. We must not call the operator + * because of strictness. */ - for (i = 0; i < mcvlist->nitems; i++) + if (item->isnull[idx] || cst->constisnull) { - bool match = true; - MCVItem *item = &mcvlist->items[i]; - - /* - * When the MCV item or the Const value is NULL we can - * treat this as a mismatch. We must not call the operator - * because of strictness. - */ - if (item->isnull[idx] || cst->constisnull) - { - matches[i] = RESULT_MERGE(matches[i], is_or, false); - continue; - } + matches[i] = RESULT_MERGE(matches[i], is_or, false); + continue; + } - /* - * Skip MCV items that can't change result in the bitmap. - * Once the value gets false for AND-lists, or true for - * OR-lists, we don't need to look at more clauses. - */ - if (RESULT_IS_FINAL(matches[i], is_or)) - continue; + /* + * Skip MCV items that can't change result in the bitmap. + * Once the value gets false for AND-lists, or true for + * OR-lists, we don't need to look at more clauses. + */ + if (RESULT_IS_FINAL(matches[i], is_or)) + continue; - /* - * First check whether the constant is below the lower - * boundary (in that case we can skip the bucket, because - * there's no overlap). - * - * We don't store collations used to build the statistics, - * but we can use the collation for the attribute itself, - * as stored in varcollid. We do reset the statistics - * after a type change (including collation change), so - * this is OK. We may need to relax this after allowing - * extended statistics on expressions. - */ - if (varonleft) - match = DatumGetBool(FunctionCall2Coll(&opproc, - var->varcollid, - item->values[idx], - cst->constvalue)); - else - match = DatumGetBool(FunctionCall2Coll(&opproc, - var->varcollid, - cst->constvalue, - item->values[idx])); - - /* update the match bitmap with the result */ - matches[i] = RESULT_MERGE(matches[i], is_or, match); - } + /* + * First check whether the constant is below the lower + * boundary (in that case we can skip the bucket, because + * there's no overlap). + * + * We don't store collations used to build the statistics, + * but we can use the collation for the attribute itself, + * as stored in varcollid. We do reset the statistics + * after a type change (including collation change), so + * this is OK. We may need to relax this after allowing + * extended statistics on expressions. + */ + if (expronleft) + match = DatumGetBool(FunctionCall2Coll(&opproc, + collid, + item->values[idx], + cst->constvalue)); + else + match = DatumGetBool(FunctionCall2Coll(&opproc, + collid, + cst->constvalue, + item->values[idx])); + + /* update the match bitmap with the result */ + matches[i] = RESULT_MERGE(matches[i], is_or, match); } } else if (IsA(clause, ScalarArrayOpExpr)) @@ -1660,115 +1743,117 @@ mcv_get_match_bitmap(PlannerInfo *root, List *clauses, ScalarArrayOpExpr *expr = (ScalarArrayOpExpr *) clause; FmgrInfo opproc; - /* valid only after examine_clause_args returns true */ - Var *var; + /* valid only after examine_opclause_args returns true */ + Node *clause_expr; Const *cst; - bool varonleft; + bool expronleft; + Oid collid; + int idx; + + /* array evaluation */ + ArrayType *arrayval; + int16 elmlen; + bool elmbyval; + char elmalign; + int num_elems; + Datum *elem_values; + bool *elem_nulls; fmgr_info(get_opcode(expr->opno), &opproc); - /* extract the var and const from the expression */ - if (examine_clause_args(expr->args, &var, &cst, &varonleft)) + /* extract the var/expr and const from the expression */ + if (!examine_opclause_args(expr->args, &clause_expr, &cst, &expronleft)) + elog(ERROR, "incompatible clause"); + + /* ScalarArrayOpExpr has the Var always on the left */ + Assert(expronleft); + + /* XXX what if (cst->constisnull == NULL)? */ + if (!cst->constisnull) { - int idx; + arrayval = DatumGetArrayTypeP(cst->constvalue); + get_typlenbyvalalign(ARR_ELEMTYPE(arrayval), + &elmlen, &elmbyval, &elmalign); + deconstruct_array(arrayval, + ARR_ELEMTYPE(arrayval), + elmlen, elmbyval, elmalign, + &elem_values, &elem_nulls, &num_elems); + } - ArrayType *arrayval; - int16 elmlen; - bool elmbyval; - char elmalign; - int num_elems; - Datum *elem_values; - bool *elem_nulls; + /* match the attribute/expression to a dimension of the statistic */ + idx = mcv_match_expression(clause_expr, keys, exprs, &collid); - /* ScalarArrayOpExpr has the Var always on the left */ - Assert(varonleft); + /* + * Walk through the MCV items and evaluate the current clause. + * We can skip items that were already ruled out, and + * terminate if there are no remaining MCV items that might + * possibly match. + */ + for (i = 0; i < mcvlist->nitems; i++) + { + int j; + bool match = (expr->useOr ? false : true); + MCVItem *item = &mcvlist->items[i]; - if (!cst->constisnull) + /* + * When the MCV item or the Const value is NULL we can + * treat this as a mismatch. We must not call the operator + * because of strictness. + */ + if (item->isnull[idx] || cst->constisnull) { - arrayval = DatumGetArrayTypeP(cst->constvalue); - get_typlenbyvalalign(ARR_ELEMTYPE(arrayval), - &elmlen, &elmbyval, &elmalign); - deconstruct_array(arrayval, - ARR_ELEMTYPE(arrayval), - elmlen, elmbyval, elmalign, - &elem_values, &elem_nulls, &num_elems); + matches[i] = RESULT_MERGE(matches[i], is_or, false); + continue; } - /* match the attribute to a dimension of the statistic */ - idx = bms_member_index(keys, var->varattno); - /* - * Walk through the MCV items and evaluate the current clause. - * We can skip items that were already ruled out, and - * terminate if there are no remaining MCV items that might - * possibly match. + * Skip MCV items that can't change result in the bitmap. + * Once the value gets false for AND-lists, or true for + * OR-lists, we don't need to look at more clauses. */ - for (i = 0; i < mcvlist->nitems; i++) + if (RESULT_IS_FINAL(matches[i], is_or)) + continue; + + for (j = 0; j < num_elems; j++) { - int j; - bool match = (expr->useOr ? false : true); - MCVItem *item = &mcvlist->items[i]; + Datum elem_value = elem_values[j]; + bool elem_isnull = elem_nulls[j]; + bool elem_match; - /* - * When the MCV item or the Const value is NULL we can - * treat this as a mismatch. We must not call the operator - * because of strictness. - */ - if (item->isnull[idx] || cst->constisnull) + /* NULL values always evaluate as not matching. */ + if (elem_isnull) { - matches[i] = RESULT_MERGE(matches[i], is_or, false); + match = RESULT_MERGE(match, expr->useOr, false); continue; } /* - * Skip MCV items that can't change result in the bitmap. - * Once the value gets false for AND-lists, or true for - * OR-lists, we don't need to look at more clauses. + * Stop evaluating the array elements once we reach + * match value that can't change - ALL() is the same + * as AND-list, ANY() is the same as OR-list. */ - if (RESULT_IS_FINAL(matches[i], is_or)) - continue; + if (RESULT_IS_FINAL(match, expr->useOr)) + break; - for (j = 0; j < num_elems; j++) - { - Datum elem_value = elem_values[j]; - bool elem_isnull = elem_nulls[j]; - bool elem_match; - - /* NULL values always evaluate as not matching. */ - if (elem_isnull) - { - match = RESULT_MERGE(match, expr->useOr, false); - continue; - } - - /* - * Stop evaluating the array elements once we reach - * match value that can't change - ALL() is the same - * as AND-list, ANY() is the same as OR-list. - */ - if (RESULT_IS_FINAL(match, expr->useOr)) - break; - - elem_match = DatumGetBool(FunctionCall2Coll(&opproc, - var->varcollid, - item->values[idx], - elem_value)); - - match = RESULT_MERGE(match, expr->useOr, elem_match); - } + elem_match = DatumGetBool(FunctionCall2Coll(&opproc, + collid, + item->values[idx], + elem_value)); - /* update the match bitmap with the result */ - matches[i] = RESULT_MERGE(matches[i], is_or, match); + match = RESULT_MERGE(match, expr->useOr, elem_match); } + + /* update the match bitmap with the result */ + matches[i] = RESULT_MERGE(matches[i], is_or, match); } } else if (IsA(clause, NullTest)) { NullTest *expr = (NullTest *) clause; - Var *var = (Var *) (expr->arg); + Node *clause_expr = (Node *) (expr->arg); - /* match the attribute to a dimension of the statistic */ - int idx = bms_member_index(keys, var->varattno); + /* match the attribute/expression to a dimension of the statistic */ + int idx = mcv_match_expression(clause_expr, keys, exprs, NULL); /* * Walk through the MCV items and evaluate the current clause. We @@ -1811,7 +1896,7 @@ mcv_get_match_bitmap(PlannerInfo *root, List *clauses, Assert(list_length(bool_clauses) >= 2); /* build the match bitmap for the OR-clauses */ - bool_matches = mcv_get_match_bitmap(root, bool_clauses, keys, + bool_matches = mcv_get_match_bitmap(root, bool_clauses, keys, exprs, mcvlist, is_orclause(clause)); /* @@ -1839,7 +1924,7 @@ mcv_get_match_bitmap(PlannerInfo *root, List *clauses, Assert(list_length(not_args) == 1); /* build the match bitmap for the NOT-clause */ - not_matches = mcv_get_match_bitmap(root, not_args, keys, + not_matches = mcv_get_match_bitmap(root, not_args, keys, exprs, mcvlist, false); /* @@ -1982,7 +2067,8 @@ mcv_clauselist_selectivity(PlannerInfo *root, StatisticExtInfo *stat, mcv = statext_mcv_load(stat->statOid); /* build a match bitmap for the clauses */ - matches = mcv_get_match_bitmap(root, clauses, stat->keys, mcv, false); + matches = mcv_get_match_bitmap(root, clauses, stat->keys, stat->exprs, + mcv, false); /* sum frequencies for all the matching MCV items */ *basesel = 0.0; @@ -2056,7 +2142,7 @@ mcv_clause_selectivity_or(PlannerInfo *root, StatisticExtInfo *stat, /* build the match bitmap for the new clause */ new_matches = mcv_get_match_bitmap(root, list_make1(clause), stat->keys, - mcv, false); + stat->exprs, mcv, false); /* * Sum the frequencies for all the MCV items matching this clause and also diff --git a/src/backend/statistics/mvdistinct.c b/src/backend/statistics/mvdistinct.c index 9ef21debb6..55d3fa0e1f 100644 --- a/src/backend/statistics/mvdistinct.c +++ b/src/backend/statistics/mvdistinct.c @@ -37,7 +37,8 @@ #include "utils/typcache.h" static double ndistinct_for_combination(double totalrows, int numrows, - HeapTuple *rows, VacAttrStats **stats, + HeapTuple *rows, ExprInfo *exprs, + int nattrs, VacAttrStats **stats, int k, int *combination); static double estimate_ndistinct(double totalrows, int numrows, int d, int f1); static int n_choose_k(int n, int k); @@ -81,16 +82,21 @@ static void generate_combinations(CombinationGenerator *state); * * This computes the ndistinct estimate using the same estimator used * in analyze.c and then computes the coefficient. + * + * To handle expressions easily, we treat them as special attributes with + * attnums above MaxHeapAttributeNumber, and we assume the expressions are + * placed after all simple attributes. */ MVNDistinct * statext_ndistinct_build(double totalrows, int numrows, HeapTuple *rows, - Bitmapset *attrs, VacAttrStats **stats) + ExprInfo *exprs, Bitmapset *attrs, + VacAttrStats **stats) { MVNDistinct *result; int k; int itemcnt; int numattrs = bms_num_members(attrs); - int numcombs = num_combinations(numattrs); + int numcombs = num_combinations(numattrs + exprs->nexprs); result = palloc(offsetof(MVNDistinct, items) + numcombs * sizeof(MVNDistinctItem)); @@ -98,14 +104,20 @@ statext_ndistinct_build(double totalrows, int numrows, HeapTuple *rows, result->type = STATS_NDISTINCT_TYPE_BASIC; result->nitems = numcombs; + /* treat expressions as special attributes with high attnums */ + attrs = add_expressions_to_attributes(attrs, exprs->nexprs); + + /* make sure there were no clashes */ + Assert(bms_num_members(attrs) == numattrs + exprs->nexprs); + itemcnt = 0; - for (k = 2; k <= numattrs; k++) + for (k = 2; k <= bms_num_members(attrs); k++) { int *combination; CombinationGenerator *generator; /* generate combinations of K out of N elements */ - generator = generator_init(numattrs, k); + generator = generator_init(bms_num_members(attrs), k); while ((combination = generator_next(generator))) { @@ -114,10 +126,32 @@ statext_ndistinct_build(double totalrows, int numrows, HeapTuple *rows, item->attrs = NULL; for (j = 0; j < k; j++) - item->attrs = bms_add_member(item->attrs, - stats[combination[j]]->attr->attnum); + { + AttrNumber attnum = InvalidAttrNumber; + + /* + * The simple attributes are before expressions, so have + * indexes below numattrs. + * */ + if (combination[j] < numattrs) + attnum = stats[combination[j]]->attr->attnum; + else + { + /* make sure the expression index is valid */ + Assert((combination[j] - numattrs) >= 0); + Assert((combination[j] - numattrs) < exprs->nexprs); + + attnum = EXPRESSION_ATTNUM(combination[j] - numattrs); + } + + Assert(attnum != InvalidAttrNumber); + + item->attrs = bms_add_member(item->attrs, attnum); + } + item->ndistinct = ndistinct_for_combination(totalrows, numrows, rows, + exprs, numattrs, stats, k, combination); itemcnt++; @@ -153,7 +187,7 @@ statext_ndistinct_load(Oid mvoid) Anum_pg_statistic_ext_data_stxdndistinct, &isnull); if (isnull) elog(ERROR, - "requested statistic kind \"%c\" is not yet built for statistics object %u", + "requested statistics kind \"%c\" is not yet built for statistics object %u", STATS_EXT_NDISTINCT, mvoid); result = statext_ndistinct_deserialize(DatumGetByteaPP(ndist)); @@ -428,6 +462,7 @@ pg_ndistinct_send(PG_FUNCTION_ARGS) */ static double ndistinct_for_combination(double totalrows, int numrows, HeapTuple *rows, + ExprInfo *exprs, int nattrs, VacAttrStats **stats, int k, int *combination) { int i, @@ -467,25 +502,57 @@ ndistinct_for_combination(double totalrows, int numrows, HeapTuple *rows, */ for (i = 0; i < k; i++) { - VacAttrStats *colstat = stats[combination[i]]; + Oid typid; TypeCacheEntry *type; + AttrNumber attnum = InvalidAttrNumber; + TupleDesc tdesc = NULL; + Oid collid = InvalidOid; + + if (combination[i] < nattrs) + { + VacAttrStats *colstat = stats[combination[i]]; + typid = colstat->attrtypid; + attnum = colstat->attr->attnum; + collid = colstat->attrcollid; + tdesc = colstat->tupDesc; + } + else + { + typid = exprs->types[combination[i] - nattrs]; + collid = exprs->collations[combination[i] - nattrs]; + } - type = lookup_type_cache(colstat->attrtypid, TYPECACHE_LT_OPR); + type = lookup_type_cache(typid, TYPECACHE_LT_OPR); if (type->lt_opr == InvalidOid) /* shouldn't happen */ elog(ERROR, "cache lookup failed for ordering operator for type %u", - colstat->attrtypid); + typid); /* prepare the sort function for this dimension */ - multi_sort_add_dimension(mss, i, type->lt_opr, colstat->attrcollid); + multi_sort_add_dimension(mss, i, type->lt_opr, collid); /* accumulate all the data for this dimension into the arrays */ for (j = 0; j < numrows; j++) { - items[j].values[i] = - heap_getattr(rows[j], - colstat->attr->attnum, - colstat->tupDesc, - &items[j].isnull[i]); + /* + * The first nattrs indexes identify simple attributes, higher + * indexes are expressions. + */ + if (combination[i] < nattrs) + items[j].values[i] = + heap_getattr(rows[j], + attnum, + tdesc, + &items[j].isnull[i]); + else + { + int idx = (combination[i] - nattrs); + + /* make sure the expression index is valid */ + Assert((idx >= 0) && (idx < exprs->nexprs)); + + items[j].values[i] = exprs->values[idx][j]; + items[j].isnull[i] = exprs->nulls[idx][j]; + } } } diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 05bb698cf4..fd69ca98cd 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -1797,7 +1797,28 @@ ProcessUtilitySlow(ParseState *pstate, break; case T_CreateStatsStmt: - address = CreateStatistics((CreateStatsStmt *) parsetree); + { + Oid relid; + CreateStatsStmt *stmt = (CreateStatsStmt *) parsetree; + RangeVar *rel = (RangeVar *) linitial(stmt->relations); + + /* + * CREATE STATISTICS will influence future execution plans + * but does not interfere with currently executing plans. + * So it should be enough to take ShareUpdateExclusiveLock + * on relation, conflicting with ANALYZE and other DDL that + * sets statistical information, but not with normal queries. + * + * XXX RangeVarCallbackOwnsRelation not needed needed here, + * to keep the same behavior as before. + */ + relid = RangeVarGetRelid(rel, ShareUpdateExclusiveLock, false); + + /* Run parse analysis ... */ + stmt = transformStatsStmt(relid, stmt, queryString); + + address = CreateStatistics(stmt); + } break; case T_AlterStatsStmt: diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c index 879288c139..bf50b32265 100644 --- a/src/backend/utils/adt/ruleutils.c +++ b/src/backend/utils/adt/ruleutils.c @@ -336,7 +336,8 @@ static char *pg_get_indexdef_worker(Oid indexrelid, int colno, bool attrsOnly, bool keysOnly, bool showTblSpc, bool inherits, int prettyFlags, bool missing_ok); -static char *pg_get_statisticsobj_worker(Oid statextid, bool missing_ok); +static char *pg_get_statisticsobj_worker(Oid statextid, bool columns_only, + bool missing_ok); static char *pg_get_partkeydef_worker(Oid relid, int prettyFlags, bool attrsOnly, bool missing_ok); static char *pg_get_constraintdef_worker(Oid constraintId, bool fullCommand, @@ -1507,7 +1508,26 @@ pg_get_statisticsobjdef(PG_FUNCTION_ARGS) Oid statextid = PG_GETARG_OID(0); char *res; - res = pg_get_statisticsobj_worker(statextid, true); + res = pg_get_statisticsobj_worker(statextid, false, true); + + if (res == NULL) + PG_RETURN_NULL(); + + PG_RETURN_TEXT_P(string_to_text(res)); +} + + +/* + * pg_get_statisticsobjdef_columns + * Get columns and expressions for an extended statistics object + */ +Datum +pg_get_statisticsobjdef_columns(PG_FUNCTION_ARGS) +{ + Oid statextid = PG_GETARG_OID(0); + char *res; + + res = pg_get_statisticsobj_worker(statextid, true, true); if (res == NULL) PG_RETURN_NULL(); @@ -1519,7 +1539,7 @@ pg_get_statisticsobjdef(PG_FUNCTION_ARGS) * Internal workhorse to decompile an extended statistics object. */ static char * -pg_get_statisticsobj_worker(Oid statextid, bool missing_ok) +pg_get_statisticsobj_worker(Oid statextid, bool columns_only, bool missing_ok) { Form_pg_statistic_ext statextrec; HeapTuple statexttup; @@ -1534,6 +1554,11 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok) bool dependencies_enabled; bool mcv_enabled; int i; + List *context; + ListCell *lc; + List *exprs = NIL; + bool has_exprs; + int ncolumns; statexttup = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(statextid)); @@ -1544,75 +1569,112 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok) elog(ERROR, "cache lookup failed for statistics object %u", statextid); } - statextrec = (Form_pg_statistic_ext) GETSTRUCT(statexttup); - - initStringInfo(&buf); + /* has the statistics expressions? */ + has_exprs = !heap_attisnull(statexttup, Anum_pg_statistic_ext_stxexprs, NULL); - nsp = get_namespace_name(statextrec->stxnamespace); - appendStringInfo(&buf, "CREATE STATISTICS %s", - quote_qualified_identifier(nsp, - NameStr(statextrec->stxname))); + statextrec = (Form_pg_statistic_ext) GETSTRUCT(statexttup); /* - * Decode the stxkind column so that we know which stats types to print. + * Get the statistics expressions, if any. (NOTE: we do not use the + * relcache versions of the expressions and predicate, because we want + * to display non-const-folded expressions.) */ - datum = SysCacheGetAttr(STATEXTOID, statexttup, - Anum_pg_statistic_ext_stxkind, &isnull); - Assert(!isnull); - arr = DatumGetArrayTypeP(datum); - if (ARR_NDIM(arr) != 1 || - ARR_HASNULL(arr) || - ARR_ELEMTYPE(arr) != CHAROID) - elog(ERROR, "stxkind is not a 1-D char array"); - enabled = (char *) ARR_DATA_PTR(arr); - - ndistinct_enabled = false; - dependencies_enabled = false; - mcv_enabled = false; - - for (i = 0; i < ARR_DIMS(arr)[0]; i++) + if (has_exprs) { - if (enabled[i] == STATS_EXT_NDISTINCT) - ndistinct_enabled = true; - if (enabled[i] == STATS_EXT_DEPENDENCIES) - dependencies_enabled = true; - if (enabled[i] == STATS_EXT_MCV) - mcv_enabled = true; + Datum exprsDatum; + bool isnull; + char *exprsString; + + exprsDatum = SysCacheGetAttr(STATEXTOID, statexttup, + Anum_pg_statistic_ext_stxexprs, &isnull); + Assert(!isnull); + exprsString = TextDatumGetCString(exprsDatum); + exprs = (List *) stringToNode(exprsString); + pfree(exprsString); } + else + exprs = NIL; - /* - * If any option is disabled, then we'll need to append the types clause - * to show which options are enabled. We omit the types clause on purpose - * when all options are enabled, so a pg_dump/pg_restore will create all - * statistics types on a newer postgres version, if the statistics had all - * options enabled on the original version. - */ - if (!ndistinct_enabled || !dependencies_enabled || !mcv_enabled) + /* count the number of columns (attributes and expressions) */ + ncolumns = statextrec->stxkeys.dim1 + list_length(exprs); + + initStringInfo(&buf); + + if (!columns_only) { - bool gotone = false; + nsp = get_namespace_name(statextrec->stxnamespace); + appendStringInfo(&buf, "CREATE STATISTICS %s", + quote_qualified_identifier(nsp, + NameStr(statextrec->stxname))); - appendStringInfoString(&buf, " ("); + /* + * Decode the stxkind column so that we know which stats types to print. + */ + datum = SysCacheGetAttr(STATEXTOID, statexttup, + Anum_pg_statistic_ext_stxkind, &isnull); + Assert(!isnull); + arr = DatumGetArrayTypeP(datum); + if (ARR_NDIM(arr) != 1 || + ARR_HASNULL(arr) || + ARR_ELEMTYPE(arr) != CHAROID) + elog(ERROR, "stxkind is not a 1-D char array"); + enabled = (char *) ARR_DATA_PTR(arr); + + ndistinct_enabled = false; + dependencies_enabled = false; + mcv_enabled = false; - if (ndistinct_enabled) + for (i = 0; i < ARR_DIMS(arr)[0]; i++) { - appendStringInfoString(&buf, "ndistinct"); - gotone = true; + if (enabled[i] == STATS_EXT_NDISTINCT) + ndistinct_enabled = true; + else if (enabled[i] == STATS_EXT_DEPENDENCIES) + dependencies_enabled = true; + else if (enabled[i] == STATS_EXT_MCV) + mcv_enabled = true; + + /* ignore STATS_EXT_EXPRESSIONS (it's built automatically) */ } - if (dependencies_enabled) + /* + * If any option is disabled, then we'll need to append the types clause + * to show which options are enabled. We omit the types clause on purpose + * when all options are enabled, so a pg_dump/pg_restore will create all + * statistics types on a newer postgres version, if the statistics had all + * options enabled on the original version. + * + * But if the statistics is defined on just a single column, it has to be + * an expression statistics. In that case we don't need to specify kinds. + */ + if ((!ndistinct_enabled || !dependencies_enabled || !mcv_enabled) && + (ncolumns > 1)) { - appendStringInfo(&buf, "%sdependencies", gotone ? ", " : ""); - gotone = true; - } + bool gotone = false; - if (mcv_enabled) - appendStringInfo(&buf, "%smcv", gotone ? ", " : ""); + appendStringInfoString(&buf, " ("); - appendStringInfoChar(&buf, ')'); - } + if (ndistinct_enabled) + { + appendStringInfoString(&buf, "ndistinct"); + gotone = true; + } + + if (dependencies_enabled) + { + appendStringInfo(&buf, "%sdependencies", gotone ? ", " : ""); + gotone = true; + } - appendStringInfoString(&buf, " ON "); + if (mcv_enabled) + appendStringInfo(&buf, "%smcv", gotone ? ", " : ""); + appendStringInfoChar(&buf, ')'); + } + + appendStringInfoString(&buf, " ON "); + } + + /* decode simple column references */ for (colno = 0; colno < statextrec->stxkeys.dim1; colno++) { AttrNumber attnum = statextrec->stxkeys.values[colno]; @@ -1626,14 +1688,109 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok) appendStringInfoString(&buf, quote_identifier(attname)); } - appendStringInfo(&buf, " FROM %s", - generate_relation_name(statextrec->stxrelid, NIL)); + context = deparse_context_for(get_relation_name(statextrec->stxrelid), + statextrec->stxrelid); + + foreach (lc, exprs) + { + Node *expr = (Node *) lfirst(lc); + char *str; + int prettyFlags = PRETTYFLAG_INDENT; + + str = deparse_expression_pretty(expr, context, false, false, + prettyFlags, 0); + + if (colno > 0) + appendStringInfoString(&buf, ", "); + + /* Need parens if it's not a bare function call */ + if (looks_like_function(expr)) + appendStringInfoString(&buf, str); + else + appendStringInfo(&buf, "(%s)", str); + + colno++; + } + + if (!columns_only) + appendStringInfo(&buf, " FROM %s", + generate_relation_name(statextrec->stxrelid, NIL)); ReleaseSysCache(statexttup); return buf.data; } +/* + * Generate text array of expressions for statistics object. + */ +Datum +pg_get_statisticsobjdef_expressions(PG_FUNCTION_ARGS) +{ + Oid statextid = PG_GETARG_OID(0); + Form_pg_statistic_ext statextrec; + HeapTuple statexttup; + Datum datum; + bool isnull; + List *context; + ListCell *lc; + List *exprs = NIL; + bool has_exprs; + char *tmp; + ArrayBuildState *astate = NULL; + + statexttup = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(statextid)); + + if (!HeapTupleIsValid(statexttup)) + elog(ERROR, "cache lookup failed for statistics object %u", statextid); + + /* has the statistics expressions? */ + has_exprs = !heap_attisnull(statexttup, Anum_pg_statistic_ext_stxexprs, NULL); + + /* no expressions? we're done */ + if (!has_exprs) + { + ReleaseSysCache(statexttup); + PG_RETURN_NULL(); + } + + statextrec = (Form_pg_statistic_ext) GETSTRUCT(statexttup); + + /* + * Get the statistics expressions, and deparse them into text values. + */ + datum = SysCacheGetAttr(STATEXTOID, statexttup, + Anum_pg_statistic_ext_stxexprs, &isnull); + + Assert(!isnull); + tmp = TextDatumGetCString(datum); + exprs = (List *) stringToNode(tmp); + pfree(tmp); + + context = deparse_context_for(get_relation_name(statextrec->stxrelid), + statextrec->stxrelid); + + foreach (lc, exprs) + { + Node *expr = (Node *) lfirst(lc); + char *str; + int prettyFlags = PRETTYFLAG_INDENT; + + str = deparse_expression_pretty(expr, context, false, false, + prettyFlags, 0); + + astate = accumArrayResult(astate, + PointerGetDatum(cstring_to_text(str)), + false, + TEXTOID, + CurrentMemoryContext); + } + + ReleaseSysCache(statexttup); + + PG_RETURN_DATUM(makeArrayResult(astate, CurrentMemoryContext)); +} + /* * pg_get_partkeydef * diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c index 52314d3aa1..a7edcaeaff 100644 --- a/src/backend/utils/adt/selfuncs.c +++ b/src/backend/utils/adt/selfuncs.c @@ -3291,6 +3291,88 @@ add_unique_group_var(PlannerInfo *root, List *varinfos, return varinfos; } +/* + * Helper routine for estimate_num_groups: add an item to a list of + * GroupExprInfos, but only if it's not known equal to any of the existing + * entries. + */ +typedef struct +{ + Node *expr; /* expression */ + RelOptInfo *rel; /* relation it belongs to */ + List *varinfos; /* info for variables in this expression */ +} GroupExprInfo; + +static List * +add_unique_group_expr(PlannerInfo *root, List *exprinfos, + Node *expr, List *vars) +{ + GroupExprInfo *exprinfo; + ListCell *lc; + Bitmapset *varnos; + Index varno; + + foreach(lc, exprinfos) + { + exprinfo = (GroupExprInfo *) lfirst(lc); + + /* Drop exact duplicates */ + if (equal(expr, exprinfo->expr)) + return exprinfos; + } + + exprinfo = (GroupExprInfo *) palloc(sizeof(GroupExprInfo)); + + varnos = pull_varnos(root, expr); + + /* + * Expressions with vars from multiple relations should never get + * here, as we split them to vars. + */ + Assert(bms_num_members(varnos) == 1); + + varno = bms_singleton_member(varnos); + + exprinfo->expr = expr; + exprinfo->varinfos = NIL; + exprinfo->rel = root->simple_rel_array[varno]; + + Assert(exprinfo->rel); + + /* Track vars for this expression. */ + foreach (lc, vars) + { + VariableStatData vardata; + Node *var = (Node *) lfirst(lc); + + /* can we get no vardata for the variable? */ + examine_variable(root, var, 0, &vardata); + + exprinfo->varinfos + = add_unique_group_var(root, exprinfo->varinfos, var, &vardata); + + ReleaseVariableStats(vardata); + } + + /* without a list of variables, use the expression itself */ + if (vars == NIL) + { + VariableStatData vardata; + + /* can we get no vardata for the variable? */ + examine_variable(root, expr, 0, &vardata); + + exprinfo->varinfos + = add_unique_group_var(root, exprinfo->varinfos, + expr, &vardata); + + ReleaseVariableStats(vardata); + } + + return lappend(exprinfos, exprinfo); +} + + /* * estimate_num_groups - Estimate number of groups in a grouped query * @@ -3360,7 +3442,7 @@ double estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows, List **pgset) { - List *varinfos = NIL; + List *exprinfos = NIL; double srf_multiplier = 1.0; double numdistinct; ListCell *l; @@ -3398,6 +3480,7 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows, double this_srf_multiplier; VariableStatData vardata; List *varshere; + Relids varnos; ListCell *l2; /* is expression in this grouping set? */ @@ -3434,8 +3517,9 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows, examine_variable(root, groupexpr, 0, &vardata); if (HeapTupleIsValid(vardata.statsTuple) || vardata.isunique) { - varinfos = add_unique_group_var(root, varinfos, - groupexpr, &vardata); + exprinfos = add_unique_group_expr(root, exprinfos, + groupexpr, NIL); + ReleaseVariableStats(vardata); continue; } @@ -3465,6 +3549,19 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows, continue; } + /* + * Are all the variables from the same relation? If yes, search for + * an extended statistic matching this expression exactly. + */ + varnos = pull_varnos(root, (Node *) varshere); + if (bms_membership(varnos) == BMS_SINGLETON) + { + exprinfos = add_unique_group_expr(root, exprinfos, + groupexpr, + varshere); + continue; + } + /* * Else add variables to varinfos list */ @@ -3472,9 +3569,8 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows, { Node *var = (Node *) lfirst(l2); - examine_variable(root, var, 0, &vardata); - varinfos = add_unique_group_var(root, varinfos, var, &vardata); - ReleaseVariableStats(vardata); + exprinfos = add_unique_group_expr(root, exprinfos, + var, NIL); } } @@ -3482,7 +3578,7 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows, * If now no Vars, we must have an all-constant or all-boolean GROUP BY * list. */ - if (varinfos == NIL) + if (exprinfos == NIL) { /* Apply SRF multiplier as we would do in the long path */ numdistinct *= srf_multiplier; @@ -3506,32 +3602,32 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows, */ do { - GroupVarInfo *varinfo1 = (GroupVarInfo *) linitial(varinfos); - RelOptInfo *rel = varinfo1->rel; + GroupExprInfo *exprinfo1 = (GroupExprInfo *) linitial(exprinfos); + RelOptInfo *rel = exprinfo1->rel; double reldistinct = 1; double relmaxndistinct = reldistinct; int relvarcount = 0; - List *newvarinfos = NIL; - List *relvarinfos = NIL; + List *newexprinfos = NIL; + List *relexprinfos = NIL; /* * Split the list of varinfos in two - one for the current rel, one * for remaining Vars on other rels. */ - relvarinfos = lappend(relvarinfos, varinfo1); - for_each_from(l, varinfos, 1) + relexprinfos = lappend(relexprinfos, exprinfo1); + for_each_from(l, exprinfos, 1) { - GroupVarInfo *varinfo2 = (GroupVarInfo *) lfirst(l); + GroupExprInfo *exprinfo2 = (GroupExprInfo *) lfirst(l); - if (varinfo2->rel == varinfo1->rel) + if (exprinfo2->rel == exprinfo1->rel) { /* varinfos on current rel */ - relvarinfos = lappend(relvarinfos, varinfo2); + relexprinfos = lappend(relexprinfos, exprinfo2); } else { - /* not time to process varinfo2 yet */ - newvarinfos = lappend(newvarinfos, varinfo2); + /* not time to process exprinfo2 yet */ + newexprinfos = lappend(newexprinfos, exprinfo2); } } @@ -3547,11 +3643,11 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows, * apply. We apply a fudge factor below, but only if we multiplied * more than one such values. */ - while (relvarinfos) + while (relexprinfos) { double mvndistinct; - if (estimate_multivariate_ndistinct(root, rel, &relvarinfos, + if (estimate_multivariate_ndistinct(root, rel, &relexprinfos, &mvndistinct)) { reldistinct *= mvndistinct; @@ -3561,18 +3657,24 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows, } else { - foreach(l, relvarinfos) + foreach(l, relexprinfos) { - GroupVarInfo *varinfo2 = (GroupVarInfo *) lfirst(l); + ListCell *lc; + GroupExprInfo *exprinfo2 = (GroupExprInfo *) lfirst(l); - reldistinct *= varinfo2->ndistinct; - if (relmaxndistinct < varinfo2->ndistinct) - relmaxndistinct = varinfo2->ndistinct; - relvarcount++; + foreach (lc, exprinfo2->varinfos) + { + GroupVarInfo *varinfo2 = (GroupVarInfo *) lfirst(lc); + + reldistinct *= varinfo2->ndistinct; + if (relmaxndistinct < varinfo2->ndistinct) + relmaxndistinct = varinfo2->ndistinct; + relvarcount++; + } } /* we're done with this relation */ - relvarinfos = NIL; + relexprinfos = NIL; } } @@ -3658,8 +3760,8 @@ estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows, numdistinct *= reldistinct; } - varinfos = newvarinfos; - } while (varinfos != NIL); + exprinfos = newexprinfos; + } while (exprinfos != NIL); /* Now we can account for the effects of any SRFs */ numdistinct *= srf_multiplier; @@ -3877,53 +3979,114 @@ estimate_hashagg_tablesize(PlannerInfo *root, Path *path, */ static bool estimate_multivariate_ndistinct(PlannerInfo *root, RelOptInfo *rel, - List **varinfos, double *ndistinct) + List **exprinfos, double *ndistinct) { ListCell *lc; - Bitmapset *attnums = NULL; - int nmatches; + int nmatches_vars; + int nmatches_exprs; Oid statOid = InvalidOid; MVNDistinct *stats; - Bitmapset *matched = NULL; + StatisticExtInfo *matched_info = NULL; /* bail out immediately if the table has no extended statistics */ if (!rel->statlist) return false; - /* Determine the attnums we're looking for */ - foreach(lc, *varinfos) - { - GroupVarInfo *varinfo = (GroupVarInfo *) lfirst(lc); - AttrNumber attnum; - - Assert(varinfo->rel == rel); - - if (!IsA(varinfo->var, Var)) - continue; - - attnum = ((Var *) varinfo->var)->varattno; - - if (!AttrNumberIsForUserDefinedAttr(attnum)) - continue; - - attnums = bms_add_member(attnums, attnum); - } - /* look for the ndistinct statistics matching the most vars */ - nmatches = 1; /* we require at least two matches */ + nmatches_vars = 0; /* we require at least two matches */ + nmatches_exprs = 0; foreach(lc, rel->statlist) { + ListCell *lc2; StatisticExtInfo *info = (StatisticExtInfo *) lfirst(lc); - Bitmapset *shared; - int nshared; + int nshared_vars = 0; + int nshared_exprs = 0; /* skip statistics of other kinds */ if (info->kind != STATS_EXT_NDISTINCT) continue; - /* compute attnums shared by the vars and the statistics object */ - shared = bms_intersect(info->keys, attnums); - nshared = bms_num_members(shared); + /* + * Determine how many expressions (and variables in non-matched + * expressions) match. We'll then use these numbers to pick the + * statistics object that best matches the clauses. + * + * XXX There's a bit of trouble with expressions - we search for + * an exact match first, and if we don't find a match we try to + * search for smaller "partial" expressions extracted from it. + * So for example given GROUP BY (a+b) we search for statistics + * defined on (a+b) first, and then maybe for one on (a) and (b). + * The trouble here is that with the current coding, the one + * matching (a) and (b) might win, because we're comparing the + * counts. We should probably give some preference to exact + * matches of the expressions. + */ + foreach(lc2, *exprinfos) + { + ListCell *lc3; + GroupExprInfo *exprinfo = (GroupExprInfo *) lfirst(lc2); + AttrNumber attnum; + bool found = false; + + Assert(exprinfo->rel == rel); + + /* simple Var, search in statistics keys directly */ + if (IsA(exprinfo->expr, Var)) + { + attnum = ((Var *) exprinfo->expr)->varattno; + + if (!AttrNumberIsForUserDefinedAttr(attnum)) + continue; + + if (bms_is_member(attnum, info->keys)) + nshared_vars++; + + continue; + } + + /* expression - see if it's in the statistics */ + foreach (lc3, info->exprs) + { + Node *expr = (Node *) lfirst(lc3); + + if (equal(exprinfo->expr, expr)) + { + nshared_exprs++; + found = true; + break; + } + } + + /* + * If it's a complex expression, and we have found it in the + * statistics object, we're done. Otherwise try to match the + * varinfos we've extracted from the expression. That way we + * can do at least some estimation. + */ + if (found) + continue; + + foreach(lc3, exprinfo->varinfos) + { + GroupVarInfo *varinfo = (GroupVarInfo *) lfirst(lc3); + + if (IsA(varinfo->var, Var)) + { + attnum = ((Var *) exprinfo->expr)->varattno; + + if (!AttrNumberIsForUserDefinedAttr(attnum)) + continue; + + if (bms_is_member(attnum, info->keys)) + nshared_vars++; + } + + /* XXX What if it's not a Var? */ + } + } + + if (nshared_vars + nshared_exprs < 2) + continue; /* * Does this statistics object match more columns than the currently @@ -3932,18 +4095,21 @@ estimate_multivariate_ndistinct(PlannerInfo *root, RelOptInfo *rel, * XXX This should break ties using name of the object, or something * like that, to make the outcome stable. */ - if (nshared > nmatches) + if ((nshared_vars > nmatches_vars) || + ((nshared_vars == nmatches_vars) && (nshared_exprs > nmatches_exprs))) { statOid = info->statOid; - nmatches = nshared; - matched = shared; + nmatches_vars = nshared_vars; + nmatches_exprs = nshared_exprs; + matched_info = info; } } /* No match? */ if (statOid == InvalidOid) return false; - Assert(nmatches > 1 && matched != NULL); + + Assert(nmatches_vars + nmatches_exprs > 1); stats = statext_ndistinct_load(statOid); @@ -3956,6 +4122,56 @@ estimate_multivariate_ndistinct(PlannerInfo *root, RelOptInfo *rel, int i; List *newlist = NIL; MVNDistinctItem *item = NULL; + ListCell *lc2; + Bitmapset *matched = NULL; + + /* see what actually matched */ + foreach (lc2, *exprinfos) + { + ListCell *lc3; + int idx; + bool found = false; + + GroupExprInfo *exprinfo = (GroupExprInfo *) lfirst(lc2); + + /* expression - see if it's in the statistics */ + idx = 0; + foreach (lc3, matched_info->exprs) + { + Node *expr = (Node *) lfirst(lc3); + + idx++; + + if (equal(exprinfo->expr, expr)) + { + matched = bms_add_member(matched, MaxHeapAttributeNumber + idx); + found = true; + break; + } + } + + if (found) + continue; + + foreach (lc3, exprinfo->varinfos) + { + GroupVarInfo *varinfo = (GroupVarInfo *) lfirst(lc2); + + /* simple Var, search in statistics keys directly */ + if (IsA(varinfo->var, Var)) + { + AttrNumber attnum = ((Var *) varinfo->var)->varattno; + + if (!AttrNumberIsForUserDefinedAttr(attnum)) + continue; + + if (!bms_is_member(attnum, matched_info->keys)) + continue; + + matched = bms_add_member(matched, attnum); + } + } + } /* Find the specific item that exactly matches the combination */ for (i = 0; i < stats->nitems; i++) @@ -3973,28 +4189,49 @@ estimate_multivariate_ndistinct(PlannerInfo *root, RelOptInfo *rel, if (!item) elog(ERROR, "corrupt MVNDistinct entry"); - /* Form the output varinfo list, keeping only unmatched ones */ - foreach(lc, *varinfos) + /* Form the output exprinfo list, keeping only unmatched ones */ + foreach(lc, *exprinfos) { - GroupVarInfo *varinfo = (GroupVarInfo *) lfirst(lc); + GroupExprInfo *exprinfo = (GroupExprInfo *) lfirst(lc); AttrNumber attnum; + ListCell *lc3; + bool found = false; + + foreach (lc3, matched_info->exprs) + { + Node *expr = (Node *) lfirst(lc3); + + if (equal(exprinfo->expr, expr)) + { + found = true; + break; + } + } - if (!IsA(varinfo->var, Var)) + /* the whole expression was matched, so skip it */ + if (found) + continue; + + if (!IsA(exprinfo->expr, Var)) { - newlist = lappend(newlist, varinfo); + /* + * FIXME Probably should remove varinfos that match the + * selected MVNDistinct item. + */ + newlist = lappend(newlist, exprinfo); continue; } - attnum = ((Var *) varinfo->var)->varattno; + attnum = ((Var *) exprinfo->expr)->varattno; if (!AttrNumberIsForUserDefinedAttr(attnum)) continue; if (!bms_is_member(attnum, matched)) - newlist = lappend(newlist, varinfo); + newlist = lappend(newlist, exprinfo); } - *varinfos = newlist; + *exprinfos = newlist; *ndistinct = item->ndistinct; return true; } @@ -4690,6 +4927,13 @@ get_join_variables(PlannerInfo *root, List *args, SpecialJoinInfo *sjinfo, *join_is_reversed = false; } +/* statext_expressions_load copies the tuple, so just pfree it. */ +static void +ReleaseDummy(HeapTuple tuple) +{ + pfree(tuple); +} + /* * examine_variable * Try to look up statistical data about an expression. @@ -4830,6 +5074,7 @@ examine_variable(PlannerInfo *root, Node *node, int varRelid, * operator we are estimating for. FIXME later. */ ListCell *ilist; + ListCell *slist; foreach(ilist, onerel->indexlist) { @@ -4986,6 +5231,68 @@ examine_variable(PlannerInfo *root, Node *node, int varRelid, if (vardata->statsTuple) break; } + + /* + * Search extended statistics for one with a matching expression. + * There might be multiple ones, so just grab the first one. In + * the future, we might consider the statistics target (and pick + * the most accurate statistics) and maybe some other parameters. + */ + foreach(slist, onerel->statlist) + { + StatisticExtInfo *info = (StatisticExtInfo *) lfirst(slist); + ListCell *expr_item; + int pos; + + /* + * Stop once we've found statistics for the expression (either + * from extended stats, or for an index in the preceding loop). + */ + if (vardata->statsTuple) + break; + + /* skip stats without per-expression stats */ + if (info->kind != STATS_EXT_EXPRESSIONS) + continue; + + pos = 0; + foreach (expr_item, info->exprs) + { + Node *expr = (Node *) lfirst(expr_item); + + Assert(expr); + + /* strip RelabelType before comparing it */ + if (expr && IsA(expr, RelabelType)) + expr = (Node *) ((RelabelType *) expr)->arg; + + /* found a match, see if we can extract pg_statistic row */ + if (equal(node, expr)) + { + HeapTuple t = statext_expressions_load(info->statOid, pos); + + vardata->statsTuple = t; + + /* + * FIXME not sure if we should cache the tuple somewhere? + * It's stored in a cached tuple in the "data" catalog, + * and we just create a new copy every time. + */ + vardata->freefunc = ReleaseDummy; + + /* + * FIXME Hack to make statistic_proc_security_check happy, + * so that this does not get rejected. Probably needs more + * thought, just a hack. + */ + vardata->acl_ok = true; + + break; + } + + pos++; + } + } } } diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..86113df29c 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2637,6 +2637,18 @@ my %tests = ( unlike => { exclude_dump_test_schema => 1, }, }, + 'CREATE STATISTICS extended_stats_expression' => { + create_order => 99, + create_sql => 'CREATE STATISTICS dump_test.test_ext_stats_expr + ON (2 * col1) FROM dump_test.test_fifth_table', + regexp => qr/^ + \QCREATE STATISTICS dump_test.test_ext_stats_expr ON ((2 * col1)) FROM dump_test.test_fifth_table;\E + /xms, + like => + { %full_runs, %dump_test_schema_runs, section_post_data => 1, }, + unlike => { exclude_dump_test_schema => 1, }, + }, + 'CREATE SEQUENCE test_table_col1_seq' => { regexp => qr/^ \QCREATE SEQUENCE dump_test.test_table_col1_seq\E diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..c1333b19d6 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -2680,15 +2680,16 @@ describeOneTableDetails(const char *schemaname, /* print any extended statistics */ if (pset.sversion >= 100000) { + /* + * FIXME this needs to be version-dependent, because older + * versions don't have pg_get_statisticsobjdef_columns. + */ printfPQExpBuffer(&buf, "SELECT oid, " "stxrelid::pg_catalog.regclass, " "stxnamespace::pg_catalog.regnamespace AS nsp, " "stxname,\n" - " (SELECT pg_catalog.string_agg(pg_catalog.quote_ident(attname),', ')\n" - " FROM pg_catalog.unnest(stxkeys) s(attnum)\n" - " JOIN pg_catalog.pg_attribute a ON (stxrelid = a.attrelid AND\n" - " a.attnum = s.attnum AND NOT attisdropped)) AS columns,\n" + "pg_get_statisticsobjdef_columns(oid) AS columns,\n" " 'd' = any(stxkind) AS ndist_enabled,\n" " 'f' = any(stxkind) AS deps_enabled,\n" " 'm' = any(stxkind) AS mcv_enabled,\n"); @@ -2715,33 +2716,60 @@ describeOneTableDetails(const char *schemaname, for (i = 0; i < tuples; i++) { bool gotone = false; + bool has_ndistinct; + bool has_dependencies; + bool has_mcv; + bool has_all; + bool has_some; + + has_ndistinct = (strcmp(PQgetvalue(result, i, 5), "t") == 0); + has_dependencies = (strcmp(PQgetvalue(result, i, 6), "t") == 0); + has_mcv = (strcmp(PQgetvalue(result, i, 7), "t") == 0); printfPQExpBuffer(&buf, " "); /* statistics object name (qualified with namespace) */ - appendPQExpBuffer(&buf, "\"%s\".\"%s\" (", + appendPQExpBuffer(&buf, "\"%s\".\"%s\"", PQgetvalue(result, i, 2), PQgetvalue(result, i, 3)); - /* options */ - if (strcmp(PQgetvalue(result, i, 5), "t") == 0) - { - appendPQExpBufferStr(&buf, "ndistinct"); - gotone = true; - } + /* + * When printing kinds we ignore expression statistics, which + * is used only internally and can't be specified by user. + * We don't print the kinds when either none are specified + * (in which case it has to be statistics on a single expr) + * or when all are specified (in which case we assume it's + * expanded by CREATE STATISTICS). + */ + has_all = (has_ndistinct && has_dependencies && has_mcv); + has_some = (has_ndistinct || has_dependencies || has_mcv); - if (strcmp(PQgetvalue(result, i, 6), "t") == 0) + if (has_some && !has_all) { - appendPQExpBuffer(&buf, "%sdependencies", gotone ? ", " : ""); - gotone = true; - } + appendPQExpBuffer(&buf, " ("); - if (strcmp(PQgetvalue(result, i, 7), "t") == 0) - { - appendPQExpBuffer(&buf, "%smcv", gotone ? ", " : ""); + /* options */ + if (has_ndistinct) + { + appendPQExpBufferStr(&buf, "ndistinct"); + gotone = true; + } + + if (has_dependencies) + { + appendPQExpBuffer(&buf, "%sdependencies", gotone ? ", " : ""); + gotone = true; + } + + if (has_mcv) + { + appendPQExpBuffer(&buf, "%smcv", gotone ? ", " : ""); + } + + appendPQExpBuffer(&buf, ")"); } - appendPQExpBuffer(&buf, ") ON %s FROM %s", + appendPQExpBuffer(&buf, " ON %s FROM %s", PQgetvalue(result, i, 4), PQgetvalue(result, i, 1)); diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 59d2b71ca9..011bd056fd 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -3655,6 +3655,14 @@ proname => 'pg_get_statisticsobjdef', provolatile => 's', prorettype => 'text', proargtypes => 'oid', prosrc => 'pg_get_statisticsobjdef' }, +{ oid => '8887', descr => 'extended statistics columns', + proname => 'pg_get_statisticsobjdef_columns', provolatile => 's', + prorettype => 'text', proargtypes => 'oid', + prosrc => 'pg_get_statisticsobjdef_columns' }, +{ oid => '8886', descr => 'extended statistics expressions', + proname => 'pg_get_statisticsobjdef_expressions', provolatile => 's', + prorettype => '_text', proargtypes => 'oid', + prosrc => 'pg_get_statisticsobjdef_expressions' }, { oid => '3352', descr => 'partition key description', proname => 'pg_get_partkeydef', provolatile => 's', prorettype => 'text', proargtypes => 'oid', prosrc => 'pg_get_partkeydef' }, diff --git a/src/include/catalog/pg_statistic_ext.h b/src/include/catalog/pg_statistic_ext.h index 29649f5814..36912ce528 100644 --- a/src/include/catalog/pg_statistic_ext.h +++ b/src/include/catalog/pg_statistic_ext.h @@ -54,6 +54,9 @@ CATALOG(pg_statistic_ext,3381,StatisticExtRelationId) #ifdef CATALOG_VARLEN char stxkind[1] BKI_FORCE_NOT_NULL; /* statistics kinds requested * to build */ + pg_node_tree stxexprs; /* A list of expression trees for stats + * attributes that are not simple column + * references. */ #endif } FormData_pg_statistic_ext; @@ -81,6 +84,7 @@ DECLARE_ARRAY_FOREIGN_KEY((stxrelid, stxkeys), pg_attribute, (attrelid, attnum)) #define STATS_EXT_NDISTINCT 'd' #define STATS_EXT_DEPENDENCIES 'f' #define STATS_EXT_MCV 'm' +#define STATS_EXT_EXPRESSIONS 'e' #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_statistic_ext_data.h b/src/include/catalog/pg_statistic_ext_data.h index 2f2577c218..9b85a5c035 100644 --- a/src/include/catalog/pg_statistic_ext_data.h +++ b/src/include/catalog/pg_statistic_ext_data.h @@ -38,6 +38,7 @@ CATALOG(pg_statistic_ext_data,3429,StatisticExtDataRelationId) pg_ndistinct stxdndistinct; /* ndistinct coefficients (serialized) */ pg_dependencies stxddependencies; /* dependencies (serialized) */ pg_mcv_list stxdmcv; /* MCV (serialized) */ + pg_statistic stxdexpr[1]; /* stats for expressions */ #endif diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h index e22df890ef..299956f329 100644 --- a/src/include/nodes/nodes.h +++ b/src/include/nodes/nodes.h @@ -454,6 +454,7 @@ typedef enum NodeTag T_TypeName, T_ColumnDef, T_IndexElem, + T_StatsElem, T_Constraint, T_DefElem, T_RangeTblEntry, diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 236832a2ca..46a9f9ee17 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -2857,8 +2857,24 @@ typedef struct CreateStatsStmt List *relations; /* rels to build stats on (list of RangeVar) */ char *stxcomment; /* comment to apply to stats, or NULL */ bool if_not_exists; /* do nothing if stats name already exists */ + bool transformed; /* true when transformStatsStmt is finished */ } CreateStatsStmt; +/* + * StatsElem - statistics parameters (used in CREATE STATISTICS) + * + * For a plain attribute, 'name' is the name of the referenced table column + * and 'expr' is NULL. For an expression, 'name' is NULL and 'expr' is the + * expression tree. + */ +typedef struct StatsElem +{ + NodeTag type; + char *name; /* name of attribute to index, or NULL */ + Node *expr; /* expression to index, or NULL */ +} StatsElem; + + /* ---------------------- * Alter Statistics Statement * ---------------------- diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index b8a6e0fc9f..e4b554f811 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -921,8 +921,9 @@ typedef struct StatisticExtInfo Oid statOid; /* OID of the statistics row */ RelOptInfo *rel; /* back-link to statistic's table */ - char kind; /* statistic kind of this entry */ + char kind; /* statistics kind of this entry */ Bitmapset *keys; /* attnums of the columns covered */ + List *exprs; /* expressions */ } StatisticExtInfo; /* diff --git a/src/include/parser/parse_node.h b/src/include/parser/parse_node.h index 176b9f37c1..a71d7e1f74 100644 --- a/src/include/parser/parse_node.h +++ b/src/include/parser/parse_node.h @@ -69,6 +69,7 @@ typedef enum ParseExprKind EXPR_KIND_FUNCTION_DEFAULT, /* default parameter value for function */ EXPR_KIND_INDEX_EXPRESSION, /* index expression */ EXPR_KIND_INDEX_PREDICATE, /* index predicate */ + EXPR_KIND_STATS_EXPRESSION, /* extended statistics expression */ EXPR_KIND_ALTER_COL_TRANSFORM, /* transform expr in ALTER COLUMN TYPE */ EXPR_KIND_EXECUTE_PARAMETER, /* parameter value in EXECUTE */ EXPR_KIND_TRIGGER_WHEN, /* WHEN condition in CREATE TRIGGER */ diff --git a/src/include/parser/parse_utilcmd.h b/src/include/parser/parse_utilcmd.h index bfa4a6b0f2..1056bf081b 100644 --- a/src/include/parser/parse_utilcmd.h +++ b/src/include/parser/parse_utilcmd.h @@ -26,6 +26,8 @@ extern AlterTableStmt *transformAlterTableStmt(Oid relid, AlterTableStmt *stmt, List **afterStmts); extern IndexStmt *transformIndexStmt(Oid relid, IndexStmt *stmt, const char *queryString); +extern CreateStatsStmt *transformStatsStmt(Oid relid, CreateStatsStmt *stmt, + const char *queryString); extern void transformRuleStmt(RuleStmt *stmt, const char *queryString, List **actions, Node **whereClause); extern List *transformCreateSchemaStmt(CreateSchemaStmt *stmt); diff --git a/src/include/statistics/extended_stats_internal.h b/src/include/statistics/extended_stats_internal.h index c849bd57c0..b2e59f9bc5 100644 --- a/src/include/statistics/extended_stats_internal.h +++ b/src/include/statistics/extended_stats_internal.h @@ -57,19 +57,35 @@ typedef struct SortItem int count; } SortItem; +/* + * Used to pass pre-computed information about expressions the stats + * object is defined on. + */ +typedef struct ExprInfo +{ + int nexprs; /* number of expressions */ + Oid *collations; /* collation for each expression */ + Oid *types; /* type of each expression */ + Datum **values; /* values for each expression */ + bool **nulls; /* nulls for each expression */ +} ExprInfo; + extern MVNDistinct *statext_ndistinct_build(double totalrows, int numrows, HeapTuple *rows, - Bitmapset *attrs, VacAttrStats **stats); + ExprInfo *exprs, Bitmapset *attrs, + VacAttrStats **stats); extern bytea *statext_ndistinct_serialize(MVNDistinct *ndistinct); extern MVNDistinct *statext_ndistinct_deserialize(bytea *data); extern MVDependencies *statext_dependencies_build(int numrows, HeapTuple *rows, - Bitmapset *attrs, VacAttrStats **stats); + ExprInfo *exprs, Bitmapset *attrs, + VacAttrStats **stats); extern bytea *statext_dependencies_serialize(MVDependencies *dependencies); extern MVDependencies *statext_dependencies_deserialize(bytea *data); extern MCVList *statext_mcv_build(int numrows, HeapTuple *rows, - Bitmapset *attrs, VacAttrStats **stats, + ExprInfo *exprs, Bitmapset *attrs, + VacAttrStats **stats, double totalrows, int stattarget); extern bytea *statext_mcv_serialize(MCVList *mcv, VacAttrStats **stats); extern MCVList *statext_mcv_deserialize(bytea *data); @@ -93,11 +109,12 @@ extern void *bsearch_arg(const void *key, const void *base, extern AttrNumber *build_attnums_array(Bitmapset *attrs, int *numattrs); extern SortItem *build_sorted_items(int numrows, int *nitems, HeapTuple *rows, - TupleDesc tdesc, MultiSortSupport mss, + ExprInfo *exprs, TupleDesc tdesc, + MultiSortSupport mss, int numattrs, AttrNumber *attnums); -extern bool examine_clause_args(List *args, Var **varp, - Const **cstp, bool *varonleftp); +extern bool examine_opclause_args(List *args, Node **exprp, + Const **cstp, bool *expronleftp); extern Selectivity mcv_combine_selectivities(Selectivity simple_sel, Selectivity mcv_sel, @@ -124,4 +141,13 @@ extern Selectivity mcv_clause_selectivity_or(PlannerInfo *root, Selectivity *overlap_basesel, Selectivity *totalsel); +extern Bitmapset *add_expressions_to_attributes(Bitmapset *attrs, int nexprs); + +/* translate 0-based expression index to attnum and back */ +#define EXPRESSION_ATTNUM(index) \ + (MaxHeapAttributeNumber + (index) + 1) + +#define EXPRESSION_INDEX(attnum) \ + ((attnum) - MaxHeapAttributeNumber - 1) + #endif /* EXTENDED_STATS_INTERNAL_H */ diff --git a/src/include/statistics/statistics.h b/src/include/statistics/statistics.h index fec50688ea..006d578e0c 100644 --- a/src/include/statistics/statistics.h +++ b/src/include/statistics/statistics.h @@ -121,6 +121,8 @@ extern Selectivity statext_clauselist_selectivity(PlannerInfo *root, extern bool has_stats_of_kind(List *stats, char requiredkind); extern StatisticExtInfo *choose_best_statistics(List *stats, char requiredkind, Bitmapset **clause_attnums, + List **clause_exprs, int nclauses); +extern HeapTuple statext_expressions_load(Oid stxoid, int idx); #endif /* STATISTICS_H */ diff --git a/src/test/regress/expected/create_table_like.out b/src/test/regress/expected/create_table_like.out index 10d17be23c..4dc5e6aa5f 100644 --- a/src/test/regress/expected/create_table_like.out +++ b/src/test/regress/expected/create_table_like.out @@ -304,7 +304,9 @@ CREATE TABLE ctlt1 (a text CHECK (length(a) > 2) PRIMARY KEY, b text); CREATE INDEX ctlt1_b_key ON ctlt1 (b); CREATE INDEX ctlt1_fnidx ON ctlt1 ((a || b)); CREATE STATISTICS ctlt1_a_b_stat ON a,b FROM ctlt1; +CREATE STATISTICS ctlt1_expr_stat ON (a || b) FROM ctlt1; COMMENT ON STATISTICS ctlt1_a_b_stat IS 'ab stats'; +COMMENT ON STATISTICS ctlt1_expr_stat IS 'ab expr stats'; COMMENT ON COLUMN ctlt1.a IS 'A'; COMMENT ON COLUMN ctlt1.b IS 'B'; COMMENT ON CONSTRAINT ctlt1_a_check ON ctlt1 IS 't1_a_check'; @@ -414,7 +416,8 @@ Indexes: Check constraints: "ctlt1_a_check" CHECK (length(a) > 2) Statistics objects: - "public"."ctlt_all_a_b_stat" (ndistinct, dependencies, mcv) ON a, b FROM ctlt_all + "public"."ctlt_all_a_b_stat" ON a, b FROM ctlt_all + "public"."ctlt_all_expr_stat" ON ((a || b)) FROM ctlt_all SELECT c.relname, objsubid, description FROM pg_description, pg_index i, pg_class c WHERE classoid = 'pg_class'::regclass AND objoid = i.indexrelid AND c.oid = i.indexrelid AND i.indrelid = 'ctlt_all'::regclass ORDER BY c.relname, objsubid; relname | objsubid | description @@ -424,10 +427,11 @@ SELECT c.relname, objsubid, description FROM pg_description, pg_index i, pg_clas (2 rows) SELECT s.stxname, objsubid, description FROM pg_description, pg_statistic_ext s WHERE classoid = 'pg_statistic_ext'::regclass AND objoid = s.oid AND s.stxrelid = 'ctlt_all'::regclass ORDER BY s.stxname, objsubid; - stxname | objsubid | description --------------------+----------+------------- - ctlt_all_a_b_stat | 0 | ab stats -(1 row) + stxname | objsubid | description +--------------------+----------+--------------- + ctlt_all_a_b_stat | 0 | ab stats + ctlt_all_expr_stat | 0 | ab expr stats +(2 rows) CREATE TABLE inh_error1 () INHERITS (ctlt1, ctlt4); NOTICE: merging multiple inherited definitions of column "a" @@ -452,7 +456,8 @@ Indexes: Check constraints: "ctlt1_a_check" CHECK (length(a) > 2) Statistics objects: - "public"."pg_attrdef_a_b_stat" (ndistinct, dependencies, mcv) ON a, b FROM public.pg_attrdef + "public"."pg_attrdef_a_b_stat" ON a, b FROM public.pg_attrdef + "public"."pg_attrdef_expr_stat" ON ((a || b)) FROM public.pg_attrdef DROP TABLE public.pg_attrdef; -- Check that LIKE isn't confused when new table masks the old, either @@ -473,7 +478,8 @@ Indexes: Check constraints: "ctlt1_a_check" CHECK (length(a) > 2) Statistics objects: - "ctl_schema"."ctlt1_a_b_stat" (ndistinct, dependencies, mcv) ON a, b FROM ctlt1 + "ctl_schema"."ctlt1_a_b_stat" ON a, b FROM ctlt1 + "ctl_schema"."ctlt1_expr_stat" ON ((a || b)) FROM ctlt1 ROLLBACK; DROP TABLE ctlt1, ctlt2, ctlt3, ctlt4, ctlt12_storage, ctlt12_comments, ctlt1_inh, ctlt13_inh, ctlt13_like, ctlt_all, ctla, ctlb CASCADE; diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out index 50d046d3ef..1461e947cd 100644 --- a/src/test/regress/expected/oidjoins.out +++ b/src/test/regress/expected/oidjoins.out @@ -151,11 +151,6 @@ NOTICE: checking pg_aggregate {aggmfinalfn} => pg_proc {oid} NOTICE: checking pg_aggregate {aggsortop} => pg_operator {oid} NOTICE: checking pg_aggregate {aggtranstype} => pg_type {oid} NOTICE: checking pg_aggregate {aggmtranstype} => pg_type {oid} -NOTICE: checking pg_statistic_ext {stxrelid} => pg_class {oid} -NOTICE: checking pg_statistic_ext {stxnamespace} => pg_namespace {oid} -NOTICE: checking pg_statistic_ext {stxowner} => pg_authid {oid} -NOTICE: checking pg_statistic_ext {stxrelid,stxkeys} => pg_attribute {attrelid,attnum} -NOTICE: checking pg_statistic_ext_data {stxoid} => pg_statistic_ext {oid} NOTICE: checking pg_statistic {starelid} => pg_class {oid} NOTICE: checking pg_statistic {staop1} => pg_operator {oid} NOTICE: checking pg_statistic {staop2} => pg_operator {oid} @@ -168,6 +163,11 @@ NOTICE: checking pg_statistic {stacoll3} => pg_collation {oid} NOTICE: checking pg_statistic {stacoll4} => pg_collation {oid} NOTICE: checking pg_statistic {stacoll5} => pg_collation {oid} NOTICE: checking pg_statistic {starelid,staattnum} => pg_attribute {attrelid,attnum} +NOTICE: checking pg_statistic_ext {stxrelid} => pg_class {oid} +NOTICE: checking pg_statistic_ext {stxnamespace} => pg_namespace {oid} +NOTICE: checking pg_statistic_ext {stxowner} => pg_authid {oid} +NOTICE: checking pg_statistic_ext {stxrelid,stxkeys} => pg_attribute {attrelid,attnum} +NOTICE: checking pg_statistic_ext_data {stxoid} => pg_statistic_ext {oid} NOTICE: checking pg_rewrite {ev_class} => pg_class {oid} NOTICE: checking pg_trigger {tgrelid} => pg_class {oid} NOTICE: checking pg_trigger {tgparentid} => pg_trigger {oid} diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index b1c9b7bdfe..1d8761775f 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -2402,6 +2402,7 @@ pg_stats_ext| SELECT cn.nspname AS schemaname, ( SELECT array_agg(a.attname ORDER BY a.attnum) AS array_agg FROM (unnest(s.stxkeys) k(k) JOIN pg_attribute a ON (((a.attrelid = s.stxrelid) AND (a.attnum = k.k))))) AS attnames, + pg_get_statisticsobjdef_expressions(s.oid) AS exprs, s.stxkind AS kinds, sd.stxdndistinct AS n_distinct, sd.stxddependencies AS dependencies, @@ -2423,6 +2424,78 @@ pg_stats_ext| SELECT cn.nspname AS schemaname, FROM (unnest(s.stxkeys) k(k) JOIN pg_attribute a ON (((a.attrelid = s.stxrelid) AND (a.attnum = k.k)))) WHERE (NOT has_column_privilege(c.oid, a.attnum, 'select'::text))))) AND ((c.relrowsecurity = false) OR (NOT row_security_active(c.oid)))); +pg_stats_ext_exprs| SELECT cn.nspname AS schemaname, + c.relname AS tablename, + sn.nspname AS statistics_schemaname, + s.stxname AS statistics_name, + pg_get_userbyid(s.stxowner) AS statistics_owner, + stat.expr, + (stat.a).stanullfrac AS null_frac, + (stat.a).stawidth AS avg_width, + (stat.a).stadistinct AS n_distinct, + CASE + WHEN ((stat.a).stakind1 = 1) THEN (stat.a).stavalues1 + WHEN ((stat.a).stakind2 = 1) THEN (stat.a).stavalues2 + WHEN ((stat.a).stakind3 = 1) THEN (stat.a).stavalues3 + WHEN ((stat.a).stakind4 = 1) THEN (stat.a).stavalues4 + WHEN ((stat.a).stakind5 = 1) THEN (stat.a).stavalues5 + ELSE NULL::anyarray + END AS most_common_vals, + CASE + WHEN ((stat.a).stakind1 = 1) THEN (stat.a).stanumbers1 + WHEN ((stat.a).stakind2 = 1) THEN (stat.a).stanumbers2 + WHEN ((stat.a).stakind3 = 1) THEN (stat.a).stanumbers3 + WHEN ((stat.a).stakind4 = 1) THEN (stat.a).stanumbers4 + WHEN ((stat.a).stakind5 = 1) THEN (stat.a).stanumbers5 + ELSE NULL::real[] + END AS most_common_freqs, + CASE + WHEN ((stat.a).stakind1 = 2) THEN (stat.a).stavalues1 + WHEN ((stat.a).stakind2 = 2) THEN (stat.a).stavalues2 + WHEN ((stat.a).stakind3 = 2) THEN (stat.a).stavalues3 + WHEN ((stat.a).stakind4 = 2) THEN (stat.a).stavalues4 + WHEN ((stat.a).stakind5 = 2) THEN (stat.a).stavalues5 + ELSE NULL::anyarray + END AS histogram_bounds, + CASE + WHEN ((stat.a).stakind1 = 3) THEN (stat.a).stanumbers1[1] + WHEN ((stat.a).stakind2 = 3) THEN (stat.a).stanumbers2[1] + WHEN ((stat.a).stakind3 = 3) THEN (stat.a).stanumbers3[1] + WHEN ((stat.a).stakind4 = 3) THEN (stat.a).stanumbers4[1] + WHEN ((stat.a).stakind5 = 3) THEN (stat.a).stanumbers5[1] + ELSE NULL::real + END AS correlation, + CASE + WHEN ((stat.a).stakind1 = 4) THEN (stat.a).stavalues1 + WHEN ((stat.a).stakind2 = 4) THEN (stat.a).stavalues2 + WHEN ((stat.a).stakind3 = 4) THEN (stat.a).stavalues3 + WHEN ((stat.a).stakind4 = 4) THEN (stat.a).stavalues4 + WHEN ((stat.a).stakind5 = 4) THEN (stat.a).stavalues5 + ELSE NULL::anyarray + END AS most_common_elems, + CASE + WHEN ((stat.a).stakind1 = 4) THEN (stat.a).stanumbers1 + WHEN ((stat.a).stakind2 = 4) THEN (stat.a).stanumbers2 + WHEN ((stat.a).stakind3 = 4) THEN (stat.a).stanumbers3 + WHEN ((stat.a).stakind4 = 4) THEN (stat.a).stanumbers4 + WHEN ((stat.a).stakind5 = 4) THEN (stat.a).stanumbers5 + ELSE NULL::real[] + END AS most_common_elem_freqs, + CASE + WHEN ((stat.a).stakind1 = 5) THEN (stat.a).stanumbers1 + WHEN ((stat.a).stakind2 = 5) THEN (stat.a).stanumbers2 + WHEN ((stat.a).stakind3 = 5) THEN (stat.a).stanumbers3 + WHEN ((stat.a).stakind4 = 5) THEN (stat.a).stanumbers4 + WHEN ((stat.a).stakind5 = 5) THEN (stat.a).stanumbers5 + ELSE NULL::real[] + END AS elem_count_histogram + FROM (((((pg_statistic_ext s + JOIN pg_class c ON ((c.oid = s.stxrelid))) + LEFT JOIN pg_statistic_ext_data sd ON ((s.oid = sd.stxoid))) + LEFT JOIN pg_namespace cn ON ((cn.oid = c.relnamespace))) + LEFT JOIN pg_namespace sn ON ((sn.oid = s.stxnamespace))) + JOIN LATERAL ( SELECT unnest(pg_get_statisticsobjdef_expressions(s.oid)) AS expr, + unnest(sd.stxdexpr) AS a) stat ON ((stat.expr IS NOT NULL))); pg_tables| SELECT n.nspname AS schemaname, c.relname AS tablename, pg_get_userbyid(c.relowner) AS tableowner, diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out index 431b3fa3de..36b7e3e7d3 100644 --- a/src/test/regress/expected/stats_ext.out +++ b/src/test/regress/expected/stats_ext.out @@ -25,7 +25,7 @@ begin end; $$; -- Verify failures -CREATE TABLE ext_stats_test (x int, y int, z int); +CREATE TABLE ext_stats_test (x text, y int, z int); CREATE STATISTICS tst; ERROR: syntax error at or near ";" LINE 1: CREATE STATISTICS tst; @@ -44,12 +44,25 @@ CREATE STATISTICS tst ON a, b FROM ext_stats_test; ERROR: column "a" does not exist CREATE STATISTICS tst ON x, x, y FROM ext_stats_test; ERROR: duplicate column name in statistics definition -CREATE STATISTICS tst ON x + y FROM ext_stats_test; -ERROR: only simple column references are allowed in CREATE STATISTICS -CREATE STATISTICS tst ON (x, y) FROM ext_stats_test; -ERROR: only simple column references are allowed in CREATE STATISTICS +CREATE STATISTICS tst ON x, x, y, x, x, y, x, x, y FROM ext_stats_test; +ERROR: cannot have more than 8 columns in statistics +CREATE STATISTICS tst ON x, x, y, x, x, (x || 'x'), (y + 1), (x || 'x'), (x || 'x'), (y + 1) FROM ext_stats_test; +ERROR: cannot have more than 8 columns in statistics +CREATE STATISTICS tst ON (x || 'x'), (x || 'x'), (y + 1), (x || 'x'), (x || 'x'), (y + 1), (x || 'x'), (x || 'x'), (y + 1) FROM ext_stats_test; +ERROR: cannot have more than 8 columns in statistics +CREATE STATISTICS tst ON (x || 'x'), (x || 'x'), y FROM ext_stats_test; +ERROR: duplicate expression in statistics definition CREATE STATISTICS tst (unrecognized) ON x, y FROM ext_stats_test; ERROR: unrecognized statistics kind "unrecognized" +-- incorrect expressions +CREATE STATISTICS tst ON y + z FROM ext_stats_test; -- missing parentheses +ERROR: syntax error at or near "+" +LINE 1: CREATE STATISTICS tst ON y + z FROM ext_stats_test; + ^ +CREATE STATISTICS tst ON (x, y) FROM ext_stats_test; -- tuple expression +ERROR: syntax error at or near "," +LINE 1: CREATE STATISTICS tst ON (x, y) FROM ext_stats_test; + ^ DROP TABLE ext_stats_test; -- Ensure stats are dropped sanely, and test IF NOT EXISTS while at it CREATE TABLE ab1 (a INTEGER, b INTEGER, c INTEGER); @@ -79,7 +92,7 @@ ALTER TABLE ab1 DROP COLUMN a; b | integer | | | c | integer | | | Statistics objects: - "public"."ab1_b_c_stats" (ndistinct, dependencies, mcv) ON b, c FROM ab1 + "public"."ab1_b_c_stats" ON b, c FROM ab1 -- Ensure statistics are dropped when table is SELECT stxname FROM pg_statistic_ext WHERE stxname LIKE 'ab1%'; @@ -111,7 +124,7 @@ ALTER STATISTICS ab1_a_b_stats SET STATISTICS 0; a | integer | | | b | integer | | | Statistics objects: - "public"."ab1_a_b_stats" (ndistinct, dependencies, mcv) ON a, b FROM ab1; STATISTICS 0 + "public"."ab1_a_b_stats" ON a, b FROM ab1; STATISTICS 0 ANALYZE ab1; SELECT stxname, stxdndistinct, stxddependencies, stxdmcv @@ -131,7 +144,7 @@ ALTER STATISTICS ab1_a_b_stats SET STATISTICS -1; a | integer | | | | plain | | b | integer | | | | plain | | Statistics objects: - "public"."ab1_a_b_stats" (ndistinct, dependencies, mcv) ON a, b FROM ab1 + "public"."ab1_a_b_stats" ON a, b FROM ab1 -- partial analyze doesn't build stats either ANALYZE ab1 (a); @@ -150,6 +163,39 @@ CREATE STATISTICS ab1_a_b_stats ON a, b FROM ab1; ANALYZE ab1; DROP TABLE ab1 CASCADE; NOTICE: drop cascades to table ab1c +-- basic test for statistics on expressions +CREATE TABLE ab1 (a INTEGER, b INTEGER, c TIMESTAMP, d TIMESTAMPTZ); +-- expression stats may be built on a single expression column +CREATE STATISTICS ab1_exprstat_1 ON (a+b) FROM ab1; +-- with a single expression, we only enable expression statistics +CREATE STATISTICS ab1_exprstat_2 ON (a+b) FROM ab1; +SELECT stxkind FROM pg_statistic_ext WHERE stxname = 'ab1_exprstat_2'; + stxkind +--------- + {e} +(1 row) + +-- adding anything to the expression builds all statistics kinds +CREATE STATISTICS ab1_exprstat_3 ON (a+b), a FROM ab1; +SELECT stxkind FROM pg_statistic_ext WHERE stxname = 'ab1_exprstat_3'; + stxkind +----------- + {d,f,m,e} +(1 row) + +-- date_trunc on timestamptz is not immutable, but that should not matter +CREATE STATISTICS ab1_exprstat_4 ON date_trunc('day', d) FROM ab1; +-- date_trunc on timestamp is immutable +CREATE STATISTICS ab1_exprstat_5 ON date_trunc('day', c) FROM ab1; +-- insert some data and run analyze, to test that these cases build properly +INSERT INTO ab1 +SELECT + generate_series(1,10), + generate_series(1,10), + generate_series('2020-10-01'::timestamp, '2020-10-10'::timestamp, interval '1 day'), + generate_series('2020-10-01'::timestamptz, '2020-10-10'::timestamptz, interval '1 day'); +ANALYZE ab1; +DROP TABLE ab1; -- Verify supported object types for extended statistics CREATE schema tststats; CREATE TABLE tststats.t (a int, b int, c text); @@ -427,6 +473,40 @@ SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE 1 | 1 (1 row) +-- a => b, a => c, b => c +TRUNCATE functional_dependencies; +DROP STATISTICS func_deps_stat; +-- now do the same thing, but with expressions +INSERT INTO functional_dependencies (a, b, c, filler1) + SELECT i, i, i, i FROM generate_series(1,5000) s(i); +ANALYZE functional_dependencies; +SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE mod(a, 11) = 1 AND mod(b::int, 13) = 1'); + estimated | actual +-----------+-------- + 1 | 35 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE mod(a, 11) = 1 AND mod(b::int, 13) = 1 AND mod(c, 7) = 1'); + estimated | actual +-----------+-------- + 1 | 5 +(1 row) + +-- create statistics +CREATE STATISTICS func_deps_stat (dependencies) ON (mod(a,11)), (mod(b::int, 13)), (mod(c, 7)) FROM functional_dependencies; +ANALYZE functional_dependencies; +SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE mod(a, 11) = 1 AND mod(b::int, 13) = 1'); + estimated | actual +-----------+-------- + 35 | 35 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE mod(a, 11) = 1 AND mod(b::int, 13) = 1 AND mod(c, 7) = 1'); + estimated | actual +-----------+-------- + 5 | 5 +(1 row) + -- a => b, a => c, b => c TRUNCATE functional_dependencies; DROP STATISTICS func_deps_stat; @@ -896,6 +976,39 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 AND b = 1 | 1 (1 row) +TRUNCATE mcv_lists; +DROP STATISTICS mcv_lists_stats; +-- random data (no MCV list), but with expression +INSERT INTO mcv_lists (a, b, c, filler1) + SELECT i, i, i, i FROM generate_series(1,5000) s(i); +ANALYZE mcv_lists; +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,37) = 1 AND mod(b::int,41) = 1'); + estimated | actual +-----------+-------- + 1 | 4 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,37) = 1 AND mod(b::int,41) = 1 AND mod(c,47) = 1'); + estimated | actual +-----------+-------- + 1 | 1 +(1 row) + +-- create statistics +CREATE STATISTICS mcv_lists_stats (mcv) ON (mod(a,37)), (mod(b::int,41)), (mod(c,47)) FROM mcv_lists; +ANALYZE mcv_lists; +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,37) = 1 AND mod(b::int,41) = 1'); + estimated | actual +-----------+-------- + 3 | 4 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,37) = 1 AND mod(b::int,41) = 1 AND mod(c,47) = 1'); + estimated | actual +-----------+-------- + 1 | 1 +(1 row) + -- 100 distinct combinations, all in the MCV list TRUNCATE mcv_lists; DROP STATISTICS mcv_lists_stats; @@ -1121,6 +1234,12 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 OR b = ' 200 | 200 (1 row) +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 OR b = ''1'' OR c = 1 OR d IS NOT NULL'); + estimated | actual +-----------+-------- + 200 | 200 +(1 row) + SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a IN (1, 2, 51, 52) AND b IN ( ''1'', ''2'')'); estimated | actual -----------+-------- @@ -1207,6 +1326,458 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 AND b = 50 | 50 (1 row) +-- 100 distinct combinations, all in the MCV list, but with expressions +TRUNCATE mcv_lists; +DROP STATISTICS mcv_lists_stats; +INSERT INTO mcv_lists (a, b, c, filler1) + SELECT i, i, i, i FROM generate_series(1,5000) s(i); +ANALYZE mcv_lists; +-- without any stats on the expressions, we have to use default selectivities, which +-- is why the estimates here are different from the pre-computed case above +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 AND mod(b::int,50) = 1'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 1 = mod(a,100) AND 1 = mod(b::int,50)'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 1 AND mod(b::int,50) < 1'); + estimated | actual +-----------+-------- + 556 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 1 > mod(a,100) AND 1 > mod(b::int,50)'); + estimated | actual +-----------+-------- + 556 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= 0 AND mod(b::int,50) <= 0'); + estimated | actual +-----------+-------- + 556 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 0 >= mod(a,100) AND 0 >= mod(b::int,50)'); + estimated | actual +-----------+-------- + 556 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 AND mod(b::int,50) = 1 AND mod(c,25) = 1'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 5 AND mod(b::int,50) < 1 AND mod(c,25) < 5'); + estimated | actual +-----------+-------- + 185 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 5 AND 1 > mod(b::int,50) AND 5 > mod(c,25)'); + estimated | actual +-----------+-------- + 185 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= 4 AND mod(b::int,50) <= 0 AND mod(c,25) <= 4'); + estimated | actual +-----------+-------- + 185 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 4 >= mod(a,100) AND 0 >= mod(b::int,50) AND 4 >= mod(c,25)'); + estimated | actual +-----------+-------- + 185 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1'); + estimated | actual +-----------+-------- + 75 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1 OR d IS NOT NULL'); + estimated | actual +-----------+-------- + 75 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) IN (1, 2, 51, 52) AND mod(b::int,50) IN ( 1, 2)'); + estimated | actual +-----------+-------- + 1 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) IN (1, 2, 51, 52, NULL) AND mod(b::int,50) IN ( 1, 2, NULL)'); + estimated | actual +-----------+-------- + 1 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = ANY (ARRAY[1, 2, 51, 52]) AND mod(b::int,50) = ANY (ARRAY[1, 2])'); + estimated | actual +-----------+-------- + 1 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = ANY (ARRAY[NULL, 1, 2, 51, 52]) AND mod(b::int,50) = ANY (ARRAY[1, 2, NULL])'); + estimated | actual +-----------+-------- + 1 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= ANY (ARRAY[1, 2, 3]) AND mod(b::int,50) IN (1, 2, 3)'); + estimated | actual +-----------+-------- + 53 | 150 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= ANY (ARRAY[1, NULL, 2, 3]) AND mod(b::int,50) IN (1, 2, NULL, 3)'); + estimated | actual +-----------+-------- + 53 | 150 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(c,25) > ANY (ARRAY[1, 2, 3])'); + estimated | actual +-----------+-------- + 391 | 100 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(c,25) > ANY (ARRAY[1, 2, 3, NULL])'); + estimated | actual +-----------+-------- + 391 | 100 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(b::int,50) IN (1, 2, 3) AND mod(c,25) > ANY (ARRAY[1, 2, 3])'); + estimated | actual +-----------+-------- + 6 | 100 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(b::int,50) IN (1, 2, NULL, 3) AND mod(c,25) > ANY (ARRAY[1, 2, NULL, 3])'); + estimated | actual +-----------+-------- + 6 | 100 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1 OR d IS NOT NULL'); + estimated | actual +-----------+-------- + 75 | 200 +(1 row) + +-- create statistics with expressions only (we create three separate stats, in order not to build more complex extended stats) +CREATE STATISTICS mcv_lists_stats_1 ON (mod(a,100)) FROM mcv_lists; +CREATE STATISTICS mcv_lists_stats_2 ON (mod(b::int,50)) FROM mcv_lists; +CREATE STATISTICS mcv_lists_stats_3 ON (mod(c,25)) FROM mcv_lists; +ANALYZE mcv_lists; +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 AND mod(b::int,50) = 1'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 1 = mod(a,100) AND 1 = mod(b::int,50)'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 1 AND mod(b::int,50) < 1'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 1 > mod(a,100) AND 1 > mod(b::int,50)'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= 0 AND mod(b::int,50) <= 0'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 0 >= mod(a,100) AND 0 >= mod(b::int,50)'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 AND mod(b::int,50) = 1 AND mod(c,25) = 1'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 5 AND mod(b::int,50) < 1 AND mod(c,25) < 5'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 5 AND 1 > mod(b::int,50) AND 5 > mod(c,25)'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= 4 AND mod(b::int,50) <= 0 AND mod(c,25) <= 4'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 4 >= mod(a,100) AND 0 >= mod(b::int,50) AND 4 >= mod(c,25)'); + estimated | actual +-----------+-------- + 1 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1'); + estimated | actual +-----------+-------- + 343 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1 OR d IS NOT NULL'); + estimated | actual +-----------+-------- + 343 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) IN (1, 2, 51, 52) AND mod(b::int,50) IN ( 1, 2)'); + estimated | actual +-----------+-------- + 8 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) IN (1, 2, 51, 52, NULL) AND mod(b::int,50) IN ( 1, 2, NULL)'); + estimated | actual +-----------+-------- + 8 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = ANY (ARRAY[1, 2, 51, 52]) AND mod(b::int,50) = ANY (ARRAY[1, 2])'); + estimated | actual +-----------+-------- + 8 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = ANY (ARRAY[NULL, 1, 2, 51, 52]) AND mod(b::int,50) = ANY (ARRAY[1, 2, NULL])'); + estimated | actual +-----------+-------- + 8 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= ANY (ARRAY[1, 2, 3]) AND mod(b::int,50) IN (1, 2, 3)'); + estimated | actual +-----------+-------- + 26 | 150 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= ANY (ARRAY[1, NULL, 2, 3]) AND mod(b::int,50) IN (1, 2, NULL, 3)'); + estimated | actual +-----------+-------- + 26 | 150 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(c,25) > ANY (ARRAY[1, 2, 3])'); + estimated | actual +-----------+-------- + 10 | 100 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(c,25) > ANY (ARRAY[1, 2, 3, NULL])'); + estimated | actual +-----------+-------- + 10 | 100 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(b::int,50) IN (1, 2, 3) AND mod(c,25) > ANY (ARRAY[1, 2, 3])'); + estimated | actual +-----------+-------- + 1 | 100 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(b::int,50) IN (1, 2, NULL, 3) AND mod(c,25) > ANY (ARRAY[1, 2, NULL, 3])'); + estimated | actual +-----------+-------- + 1 | 100 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1 OR d IS NOT NULL'); + estimated | actual +-----------+-------- + 343 | 200 +(1 row) + +DROP STATISTICS mcv_lists_stats_1; +DROP STATISTICS mcv_lists_stats_2; +DROP STATISTICS mcv_lists_stats_3; +-- create statistics with both MCV and expressions +CREATE STATISTICS mcv_lists_stats (mcv) ON (mod(a,100)), (mod(b::int,50)), (mod(c,25)) FROM mcv_lists; +ANALYZE mcv_lists; +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 AND mod(b::int,50) = 1'); + estimated | actual +-----------+-------- + 50 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 1 = mod(a,100) AND 1 = mod(b::int,50)'); + estimated | actual +-----------+-------- + 50 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 1 AND mod(b::int,50) < 1'); + estimated | actual +-----------+-------- + 50 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 1 > mod(a,100) AND 1 > mod(b::int,50)'); + estimated | actual +-----------+-------- + 50 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= 0 AND mod(b::int,50) <= 0'); + estimated | actual +-----------+-------- + 50 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 0 >= mod(a,100) AND 0 >= mod(b::int,50)'); + estimated | actual +-----------+-------- + 50 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 AND mod(b::int,50) = 1 AND mod(c,25) = 1'); + estimated | actual +-----------+-------- + 50 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 5 AND mod(b::int,50) < 1 AND mod(c,25) < 5'); + estimated | actual +-----------+-------- + 50 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 5 AND 1 > mod(b::int,50) AND 5 > mod(c,25)'); + estimated | actual +-----------+-------- + 50 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= 4 AND mod(b::int,50) <= 0 AND mod(c,25) <= 4'); + estimated | actual +-----------+-------- + 50 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 4 >= mod(a,100) AND 0 >= mod(b::int,50) AND 4 >= mod(c,25)'); + estimated | actual +-----------+-------- + 50 | 50 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1'); + estimated | actual +-----------+-------- + 200 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1 OR d IS NOT NULL'); + estimated | actual +-----------+-------- + 200 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) IN (1, 2, 51, 52) AND mod(b::int,50) IN ( 1, 2)'); + estimated | actual +-----------+-------- + 200 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) IN (1, 2, 51, 52, NULL) AND mod(b::int,50) IN ( 1, 2, NULL)'); + estimated | actual +-----------+-------- + 200 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = ANY (ARRAY[1, 2, 51, 52]) AND mod(b::int,50) = ANY (ARRAY[1, 2])'); + estimated | actual +-----------+-------- + 200 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = ANY (ARRAY[NULL, 1, 2, 51, 52]) AND mod(b::int,50) = ANY (ARRAY[1, 2, NULL])'); + estimated | actual +-----------+-------- + 200 | 200 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= ANY (ARRAY[1, 2, 3]) AND mod(b::int,50) IN (1, 2, 3)'); + estimated | actual +-----------+-------- + 150 | 150 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= ANY (ARRAY[1, NULL, 2, 3]) AND mod(b::int,50) IN (1, 2, NULL, 3)'); + estimated | actual +-----------+-------- + 150 | 150 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(c,25) > ANY (ARRAY[1, 2, 3])'); + estimated | actual +-----------+-------- + 100 | 100 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(c,25) > ANY (ARRAY[1, 2, 3, NULL])'); + estimated | actual +-----------+-------- + 100 | 100 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(b::int,50) IN (1, 2, 3) AND mod(c,25) > ANY (ARRAY[1, 2, 3])'); + estimated | actual +-----------+-------- + 100 | 100 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(b::int,50) IN (1, 2, NULL, 3) AND mod(c,25) > ANY (ARRAY[1, 2, NULL, 3])'); + estimated | actual +-----------+-------- + 100 | 100 +(1 row) + +-- we can't use the statistic for OR clauses that are not fully covered (missing 'd' attribute) +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1 OR d IS NOT NULL'); + estimated | actual +-----------+-------- + 200 | 200 +(1 row) + -- 100 distinct combinations with NULL values, all in the MCV list TRUNCATE mcv_lists; DROP STATISTICS mcv_lists_stats; @@ -1712,6 +2283,100 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_multi WHERE a = 0 OR (1 row) DROP TABLE mcv_lists_multi; +-- statistics on integer expressions +CREATE TABLE expr_stats (a int, b int, c int); +INSERT INTO expr_stats SELECT mod(i,10), mod(i,10), mod(i,10) FROM generate_series(1,10000) s(i); +ANALYZE expr_stats; +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE (2*a) = 0 AND (3*b) = 0'); + estimated | actual +-----------+-------- + 1 | 1000 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE (a+b) = 0 AND (a-b) = 0'); + estimated | actual +-----------+-------- + 1 | 1000 +(1 row) + +CREATE STATISTICS expr_stats_1 (mcv) ON (a+b), (a-b), (2*a), (3*b) FROM expr_stats; +ANALYZE expr_stats; +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE (2*a) = 0 AND (3*b) = 0'); + estimated | actual +-----------+-------- + 1000 | 1000 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE (a+b) = 0 AND (a-b) = 0'); + estimated | actual +-----------+-------- + 1000 | 1000 +(1 row) + +DROP STATISTICS expr_stats_1; +DROP TABLE expr_stats; +-- statistics on a mix columns and expressions +CREATE TABLE expr_stats (a int, b int, c int); +INSERT INTO expr_stats SELECT mod(i,10), mod(i,10), mod(i,10) FROM generate_series(1,10000) s(i); +ANALYZE expr_stats; +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND (2*a) = 0 AND (3*b) = 0'); + estimated | actual +-----------+-------- + 1 | 1000 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 3 AND b = 3 AND (a-b) = 0'); + estimated | actual +-----------+-------- + 1 | 1000 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND b = 1 AND (a-b) = 0'); + estimated | actual +-----------+-------- + 1 | 0 +(1 row) + +CREATE STATISTICS expr_stats_1 (mcv) ON a, b, (2*a), (3*b), (a+b), (a-b) FROM expr_stats; +ANALYZE expr_stats; +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND (2*a) = 0 AND (3*b) = 0'); + estimated | actual +-----------+-------- + 1000 | 1000 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 3 AND b = 3 AND (a-b) = 0'); + estimated | actual +-----------+-------- + 1000 | 1000 +(1 row) + +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND b = 1 AND (a-b) = 0'); + estimated | actual +-----------+-------- + 1 | 0 +(1 row) + +DROP TABLE expr_stats; +-- statistics on expressions with different data types +CREATE TABLE expr_stats (a int, b name, c text); +INSERT INTO expr_stats SELECT mod(i,10), md5(mod(i,10)::text), md5(mod(i,10)::text) FROM generate_series(1,10000) s(i); +ANALYZE expr_stats; +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND (b || c) <= ''z'' AND (c || b) >= ''0'''); + estimated | actual +-----------+-------- + 111 | 1000 +(1 row) + +CREATE STATISTICS expr_stats_1 (mcv) ON a, b, (b || c), (c || b) FROM expr_stats; +ANALYZE expr_stats; +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND (b || c) <= ''z'' AND (c || b) >= ''0'''); + estimated | actual +-----------+-------- + 1000 | 1000 +(1 row) + +DROP TABLE expr_stats; -- Permission tests. Users should not be able to see specific data values in -- the extended statistics, if they lack permission to see those values in -- the underlying table. diff --git a/src/test/regress/sql/create_table_like.sql b/src/test/regress/sql/create_table_like.sql index 06b76f949d..4929d373a2 100644 --- a/src/test/regress/sql/create_table_like.sql +++ b/src/test/regress/sql/create_table_like.sql @@ -124,7 +124,9 @@ CREATE TABLE ctlt1 (a text CHECK (length(a) > 2) PRIMARY KEY, b text); CREATE INDEX ctlt1_b_key ON ctlt1 (b); CREATE INDEX ctlt1_fnidx ON ctlt1 ((a || b)); CREATE STATISTICS ctlt1_a_b_stat ON a,b FROM ctlt1; +CREATE STATISTICS ctlt1_expr_stat ON (a || b) FROM ctlt1; COMMENT ON STATISTICS ctlt1_a_b_stat IS 'ab stats'; +COMMENT ON STATISTICS ctlt1_expr_stat IS 'ab expr stats'; COMMENT ON COLUMN ctlt1.a IS 'A'; COMMENT ON COLUMN ctlt1.b IS 'B'; COMMENT ON CONSTRAINT ctlt1_a_check ON ctlt1 IS 't1_a_check'; diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql index 0d7a114b19..bd2ada1676 100644 --- a/src/test/regress/sql/stats_ext.sql +++ b/src/test/regress/sql/stats_ext.sql @@ -28,16 +28,21 @@ end; $$; -- Verify failures -CREATE TABLE ext_stats_test (x int, y int, z int); +CREATE TABLE ext_stats_test (x text, y int, z int); CREATE STATISTICS tst; CREATE STATISTICS tst ON a, b; CREATE STATISTICS tst FROM sometab; CREATE STATISTICS tst ON a, b FROM nonexistent; CREATE STATISTICS tst ON a, b FROM ext_stats_test; CREATE STATISTICS tst ON x, x, y FROM ext_stats_test; -CREATE STATISTICS tst ON x + y FROM ext_stats_test; -CREATE STATISTICS tst ON (x, y) FROM ext_stats_test; +CREATE STATISTICS tst ON x, x, y, x, x, y, x, x, y FROM ext_stats_test; +CREATE STATISTICS tst ON x, x, y, x, x, (x || 'x'), (y + 1), (x || 'x'), (x || 'x'), (y + 1) FROM ext_stats_test; +CREATE STATISTICS tst ON (x || 'x'), (x || 'x'), (y + 1), (x || 'x'), (x || 'x'), (y + 1), (x || 'x'), (x || 'x'), (y + 1) FROM ext_stats_test; +CREATE STATISTICS tst ON (x || 'x'), (x || 'x'), y FROM ext_stats_test; CREATE STATISTICS tst (unrecognized) ON x, y FROM ext_stats_test; +-- incorrect expressions +CREATE STATISTICS tst ON y + z FROM ext_stats_test; -- missing parentheses +CREATE STATISTICS tst ON (x, y) FROM ext_stats_test; -- tuple expression DROP TABLE ext_stats_test; -- Ensure stats are dropped sanely, and test IF NOT EXISTS while at it @@ -97,6 +102,36 @@ CREATE STATISTICS ab1_a_b_stats ON a, b FROM ab1; ANALYZE ab1; DROP TABLE ab1 CASCADE; +-- basic test for statistics on expressions +CREATE TABLE ab1 (a INTEGER, b INTEGER, c TIMESTAMP, d TIMESTAMPTZ); + +-- expression stats may be built on a single expression column +CREATE STATISTICS ab1_exprstat_1 ON (a+b) FROM ab1; + +-- with a single expression, we only enable expression statistics +CREATE STATISTICS ab1_exprstat_2 ON (a+b) FROM ab1; +SELECT stxkind FROM pg_statistic_ext WHERE stxname = 'ab1_exprstat_2'; + +-- adding anything to the expression builds all statistics kinds +CREATE STATISTICS ab1_exprstat_3 ON (a+b), a FROM ab1; +SELECT stxkind FROM pg_statistic_ext WHERE stxname = 'ab1_exprstat_3'; + +-- date_trunc on timestamptz is not immutable, but that should not matter +CREATE STATISTICS ab1_exprstat_4 ON date_trunc('day', d) FROM ab1; + +-- date_trunc on timestamp is immutable +CREATE STATISTICS ab1_exprstat_5 ON date_trunc('day', c) FROM ab1; + +-- insert some data and run analyze, to test that these cases build properly +INSERT INTO ab1 +SELECT + generate_series(1,10), + generate_series(1,10), + generate_series('2020-10-01'::timestamp, '2020-10-10'::timestamp, interval '1 day'), + generate_series('2020-10-01'::timestamptz, '2020-10-10'::timestamptz, interval '1 day'); +ANALYZE ab1; +DROP TABLE ab1; + -- Verify supported object types for extended statistics CREATE schema tststats; @@ -272,6 +307,29 @@ SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE TRUNCATE functional_dependencies; DROP STATISTICS func_deps_stat; +-- now do the same thing, but with expressions +INSERT INTO functional_dependencies (a, b, c, filler1) + SELECT i, i, i, i FROM generate_series(1,5000) s(i); + +ANALYZE functional_dependencies; + +SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE mod(a, 11) = 1 AND mod(b::int, 13) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE mod(a, 11) = 1 AND mod(b::int, 13) = 1 AND mod(c, 7) = 1'); + +-- create statistics +CREATE STATISTICS func_deps_stat (dependencies) ON (mod(a,11)), (mod(b::int, 13)), (mod(c, 7)) FROM functional_dependencies; + +ANALYZE functional_dependencies; + +SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE mod(a, 11) = 1 AND mod(b::int, 13) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM functional_dependencies WHERE mod(a, 11) = 1 AND mod(b::int, 13) = 1 AND mod(c, 7) = 1'); + +-- a => b, a => c, b => c +TRUNCATE functional_dependencies; +DROP STATISTICS func_deps_stat; + INSERT INTO functional_dependencies (a, b, c, filler1) SELECT mod(i,100), mod(i,50), mod(i,25), i FROM generate_series(1,5000) s(i); @@ -479,6 +537,28 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 AND b = SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 AND b = ''1'' AND c = 1'); +TRUNCATE mcv_lists; +DROP STATISTICS mcv_lists_stats; + +-- random data (no MCV list), but with expression +INSERT INTO mcv_lists (a, b, c, filler1) + SELECT i, i, i, i FROM generate_series(1,5000) s(i); + +ANALYZE mcv_lists; + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,37) = 1 AND mod(b::int,41) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,37) = 1 AND mod(b::int,41) = 1 AND mod(c,47) = 1'); + +-- create statistics +CREATE STATISTICS mcv_lists_stats (mcv) ON (mod(a,37)), (mod(b::int,41)), (mod(c,47)) FROM mcv_lists; + +ANALYZE mcv_lists; + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,37) = 1 AND mod(b::int,41) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,37) = 1 AND mod(b::int,41) = 1 AND mod(c,47) = 1'); + -- 100 distinct combinations, all in the MCV list TRUNCATE mcv_lists; DROP STATISTICS mcv_lists_stats; @@ -565,6 +645,8 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 OR b = ' SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 OR b = ''1'' OR c = 1 OR d IS NOT NULL'); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 OR b = ''1'' OR c = 1 OR d IS NOT NULL'); + SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a IN (1, 2, 51, 52) AND b IN ( ''1'', ''2'')'); SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a IN (1, 2, 51, 52, NULL) AND b IN ( ''1'', ''2'', NULL)'); @@ -602,6 +684,180 @@ ANALYZE mcv_lists; SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 AND b = ''1'''); + +-- 100 distinct combinations, all in the MCV list, but with expressions +TRUNCATE mcv_lists; +DROP STATISTICS mcv_lists_stats; + +INSERT INTO mcv_lists (a, b, c, filler1) + SELECT i, i, i, i FROM generate_series(1,5000) s(i); + +ANALYZE mcv_lists; + +-- without any stats on the expressions, we have to use default selectivities, which +-- is why the estimates here are different from the pre-computed case above + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 AND mod(b::int,50) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 1 = mod(a,100) AND 1 = mod(b::int,50)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 1 AND mod(b::int,50) < 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 1 > mod(a,100) AND 1 > mod(b::int,50)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= 0 AND mod(b::int,50) <= 0'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 0 >= mod(a,100) AND 0 >= mod(b::int,50)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 AND mod(b::int,50) = 1 AND mod(c,25) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 5 AND mod(b::int,50) < 1 AND mod(c,25) < 5'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 5 AND 1 > mod(b::int,50) AND 5 > mod(c,25)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= 4 AND mod(b::int,50) <= 0 AND mod(c,25) <= 4'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 4 >= mod(a,100) AND 0 >= mod(b::int,50) AND 4 >= mod(c,25)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1 OR d IS NOT NULL'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) IN (1, 2, 51, 52) AND mod(b::int,50) IN ( 1, 2)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) IN (1, 2, 51, 52, NULL) AND mod(b::int,50) IN ( 1, 2, NULL)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = ANY (ARRAY[1, 2, 51, 52]) AND mod(b::int,50) = ANY (ARRAY[1, 2])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = ANY (ARRAY[NULL, 1, 2, 51, 52]) AND mod(b::int,50) = ANY (ARRAY[1, 2, NULL])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= ANY (ARRAY[1, 2, 3]) AND mod(b::int,50) IN (1, 2, 3)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= ANY (ARRAY[1, NULL, 2, 3]) AND mod(b::int,50) IN (1, 2, NULL, 3)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(c,25) > ANY (ARRAY[1, 2, 3])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(c,25) > ANY (ARRAY[1, 2, 3, NULL])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(b::int,50) IN (1, 2, 3) AND mod(c,25) > ANY (ARRAY[1, 2, 3])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(b::int,50) IN (1, 2, NULL, 3) AND mod(c,25) > ANY (ARRAY[1, 2, NULL, 3])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1 OR d IS NOT NULL'); + +-- create statistics with expressions only (we create three separate stats, in order not to build more complex extended stats) +CREATE STATISTICS mcv_lists_stats_1 ON (mod(a,100)) FROM mcv_lists; +CREATE STATISTICS mcv_lists_stats_2 ON (mod(b::int,50)) FROM mcv_lists; +CREATE STATISTICS mcv_lists_stats_3 ON (mod(c,25)) FROM mcv_lists; + +ANALYZE mcv_lists; + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 AND mod(b::int,50) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 1 = mod(a,100) AND 1 = mod(b::int,50)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 1 AND mod(b::int,50) < 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 1 > mod(a,100) AND 1 > mod(b::int,50)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= 0 AND mod(b::int,50) <= 0'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 0 >= mod(a,100) AND 0 >= mod(b::int,50)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 AND mod(b::int,50) = 1 AND mod(c,25) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 5 AND mod(b::int,50) < 1 AND mod(c,25) < 5'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 5 AND 1 > mod(b::int,50) AND 5 > mod(c,25)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= 4 AND mod(b::int,50) <= 0 AND mod(c,25) <= 4'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 4 >= mod(a,100) AND 0 >= mod(b::int,50) AND 4 >= mod(c,25)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1 OR d IS NOT NULL'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) IN (1, 2, 51, 52) AND mod(b::int,50) IN ( 1, 2)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) IN (1, 2, 51, 52, NULL) AND mod(b::int,50) IN ( 1, 2, NULL)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = ANY (ARRAY[1, 2, 51, 52]) AND mod(b::int,50) = ANY (ARRAY[1, 2])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = ANY (ARRAY[NULL, 1, 2, 51, 52]) AND mod(b::int,50) = ANY (ARRAY[1, 2, NULL])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= ANY (ARRAY[1, 2, 3]) AND mod(b::int,50) IN (1, 2, 3)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= ANY (ARRAY[1, NULL, 2, 3]) AND mod(b::int,50) IN (1, 2, NULL, 3)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(c,25) > ANY (ARRAY[1, 2, 3])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(c,25) > ANY (ARRAY[1, 2, 3, NULL])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(b::int,50) IN (1, 2, 3) AND mod(c,25) > ANY (ARRAY[1, 2, 3])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(b::int,50) IN (1, 2, NULL, 3) AND mod(c,25) > ANY (ARRAY[1, 2, NULL, 3])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1 OR d IS NOT NULL'); + +DROP STATISTICS mcv_lists_stats_1; +DROP STATISTICS mcv_lists_stats_2; +DROP STATISTICS mcv_lists_stats_3; + +-- create statistics with both MCV and expressions +CREATE STATISTICS mcv_lists_stats (mcv) ON (mod(a,100)), (mod(b::int,50)), (mod(c,25)) FROM mcv_lists; + +ANALYZE mcv_lists; + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 AND mod(b::int,50) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 1 = mod(a,100) AND 1 = mod(b::int,50)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 1 AND mod(b::int,50) < 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 1 > mod(a,100) AND 1 > mod(b::int,50)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= 0 AND mod(b::int,50) <= 0'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 0 >= mod(a,100) AND 0 >= mod(b::int,50)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 AND mod(b::int,50) = 1 AND mod(c,25) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 5 AND mod(b::int,50) < 1 AND mod(c,25) < 5'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < 5 AND 1 > mod(b::int,50) AND 5 > mod(c,25)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= 4 AND mod(b::int,50) <= 0 AND mod(c,25) <= 4'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE 4 >= mod(a,100) AND 0 >= mod(b::int,50) AND 4 >= mod(c,25)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1 OR d IS NOT NULL'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) IN (1, 2, 51, 52) AND mod(b::int,50) IN ( 1, 2)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) IN (1, 2, 51, 52, NULL) AND mod(b::int,50) IN ( 1, 2, NULL)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = ANY (ARRAY[1, 2, 51, 52]) AND mod(b::int,50) = ANY (ARRAY[1, 2])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = ANY (ARRAY[NULL, 1, 2, 51, 52]) AND mod(b::int,50) = ANY (ARRAY[1, 2, NULL])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= ANY (ARRAY[1, 2, 3]) AND mod(b::int,50) IN (1, 2, 3)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) <= ANY (ARRAY[1, NULL, 2, 3]) AND mod(b::int,50) IN (1, 2, NULL, 3)'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(c,25) > ANY (ARRAY[1, 2, 3])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(c,25) > ANY (ARRAY[1, 2, 3, NULL])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(b::int,50) IN (1, 2, 3) AND mod(c,25) > ANY (ARRAY[1, 2, 3])'); + +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) < ALL (ARRAY[4, 5]) AND mod(b::int,50) IN (1, 2, NULL, 3) AND mod(c,25) > ANY (ARRAY[1, 2, NULL, 3])'); + +-- we can't use the statistic for OR clauses that are not fully covered (missing 'd' attribute) +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE mod(a,100) = 1 OR mod(b::int,50) = 1 OR mod(c,25) = 1 OR d IS NOT NULL'); + -- 100 distinct combinations with NULL values, all in the MCV list TRUNCATE mcv_lists; DROP STATISTICS mcv_lists_stats; @@ -894,6 +1150,57 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_multi WHERE a = 0 OR DROP TABLE mcv_lists_multi; + +-- statistics on integer expressions +CREATE TABLE expr_stats (a int, b int, c int); +INSERT INTO expr_stats SELECT mod(i,10), mod(i,10), mod(i,10) FROM generate_series(1,10000) s(i); +ANALYZE expr_stats; + +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE (2*a) = 0 AND (3*b) = 0'); +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE (a+b) = 0 AND (a-b) = 0'); + +CREATE STATISTICS expr_stats_1 (mcv) ON (a+b), (a-b), (2*a), (3*b) FROM expr_stats; +ANALYZE expr_stats; + +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE (2*a) = 0 AND (3*b) = 0'); +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE (a+b) = 0 AND (a-b) = 0'); + +DROP STATISTICS expr_stats_1; +DROP TABLE expr_stats; + +-- statistics on a mix columns and expressions +CREATE TABLE expr_stats (a int, b int, c int); +INSERT INTO expr_stats SELECT mod(i,10), mod(i,10), mod(i,10) FROM generate_series(1,10000) s(i); +ANALYZE expr_stats; + +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND (2*a) = 0 AND (3*b) = 0'); +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 3 AND b = 3 AND (a-b) = 0'); +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND b = 1 AND (a-b) = 0'); + +CREATE STATISTICS expr_stats_1 (mcv) ON a, b, (2*a), (3*b), (a+b), (a-b) FROM expr_stats; +ANALYZE expr_stats; + +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND (2*a) = 0 AND (3*b) = 0'); +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 3 AND b = 3 AND (a-b) = 0'); +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND b = 1 AND (a-b) = 0'); + +DROP TABLE expr_stats; + +-- statistics on expressions with different data types +CREATE TABLE expr_stats (a int, b name, c text); +INSERT INTO expr_stats SELECT mod(i,10), md5(mod(i,10)::text), md5(mod(i,10)::text) FROM generate_series(1,10000) s(i); +ANALYZE expr_stats; + +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND (b || c) <= ''z'' AND (c || b) >= ''0'''); + +CREATE STATISTICS expr_stats_1 (mcv) ON a, b, (b || c), (c || b) FROM expr_stats; +ANALYZE expr_stats; + +SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND (b || c) <= ''z'' AND (c || b) >= ''0'''); + +DROP TABLE expr_stats; + + -- Permission tests. Users should not be able to see specific data values in -- the extended statistics, if they lack permission to see those values in -- the underlying table. -- 2.26.2 --------------614DDB87AFFED893713AC0E9 Content-Type: text/x-patch; charset=UTF-8; name="0004-WIP-rework-tracking-of-expressions-20210304.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0004-WIP-rework-tracking-of-expressions-20210304.patch" ^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Allow tests to pass in OpenSSL FIPS mode @ 2023-01-31 09:55 Peter Eisentraut <[email protected]> 0 siblings, 1 reply; 6+ messages in thread From: Peter Eisentraut @ 2023-01-31 09:55 UTC (permalink / raw) To: Michael Paquier <[email protected]>; +Cc: pgsql-hackers On 09.12.22 05:16, Michael Paquier wrote: >> Some tests inspect the actual md5 result strings or build statistics based >> on them. I have tried to carefully preserve the meaning of the original >> tests, to the extent that they could be inferred, in some cases adjusting >> example values by matching the md5 outputs to the equivalent sha256 outputs. >> Some cases are tricky or mysterious or both and could use another look. > incremental_sort mostly relies on the plan generated, so the change > should be rather straight-forward I guess, though there may be a side > effect depending on costing. Hmm, it does not look like stats_ext > would be an issue as it checks the stats correlation of the attributes > for mcv_lists_arrays. > > largeobject_1.out has been forgotten in the set requiring a refresh. Here is a refreshed patch with the missing file added. From 28a32d41bf93c682caba2bacd94bee0f389915da Mon Sep 17 00:00:00 2001 From: Peter Eisentraut <[email protected]> Date: Wed, 7 Dec 2022 13:32:26 +0100 Subject: [PATCH v2] Remove incidental md5() function uses from main regression tests Most of these calls were to generate some random data. These can be replaced by appropriately adapted sha256() calls. This will eventually allow these tests to pass in OpenSSL FIPS mode (which does not allow MD5 use). Similar work for other test suites will follow later. Discussion: https://www.postgresql.org/message-id/flat/[email protected] --- src/test/regress/expected/arrays.out | 18 +- src/test/regress/expected/brin.out | 4 +- src/test/regress/expected/brin_multi.out | 8 +- src/test/regress/expected/compression.out | 13 +- src/test/regress/expected/compression_1.out | 11 +- src/test/regress/expected/inherit.out | 2 +- src/test/regress/expected/largeobject.out | 2 +- src/test/regress/expected/largeobject_1.out | 2 +- src/test/regress/expected/matview.out | 8 +- src/test/regress/expected/memoize.out | 2 +- src/test/regress/expected/plpgsql.out | 24 +- src/test/regress/expected/rowsecurity.out | 591 ++++++++++---------- src/test/regress/expected/stats_ext.out | 14 +- src/test/regress/sql/arrays.sql | 18 +- src/test/regress/sql/brin.sql | 4 +- src/test/regress/sql/brin_multi.sql | 8 +- src/test/regress/sql/compression.sql | 7 +- src/test/regress/sql/inherit.sql | 2 +- src/test/regress/sql/largeobject.sql | 2 +- src/test/regress/sql/matview.sql | 8 +- src/test/regress/sql/memoize.sql | 2 +- src/test/regress/sql/plpgsql.sql | 2 +- src/test/regress/sql/rowsecurity.sql | 14 +- src/test/regress/sql/stats_ext.sql | 14 +- 24 files changed, 387 insertions(+), 393 deletions(-) diff --git a/src/test/regress/expected/arrays.out b/src/test/regress/expected/arrays.out index a2f9d7ed16..c6c084b088 100644 --- a/src/test/regress/expected/arrays.out +++ b/src/test/regress/expected/arrays.out @@ -2297,20 +2297,20 @@ select * from t1; (1 row) -- Check that arrays of composites are safely detoasted when needed -create temp table src (f1 text); +create temp table src (f1 bytea); insert into src - select string_agg(random()::text,'') from generate_series(1,10000); -create type textandtext as (c1 text, c2 text); -create temp table dest (f1 textandtext[]); -insert into dest select array[row(f1,f1)::textandtext] from src; -select length(md5((f1[1]).c2)) from dest; + select string_agg(random()::text::bytea,'') from generate_series(1,10000); +create type byteaandbytea as (c1 bytea, c2 bytea); +create temp table dest (f1 byteaandbytea[]); +insert into dest select array[row(f1,f1)::byteaandbytea] from src; +select length(sha256((f1[1]).c2)) from dest; length -------- 32 (1 row) delete from src; -select length(md5((f1[1]).c2)) from dest; +select length(sha256((f1[1]).c2)) from dest; length -------- 32 @@ -2318,14 +2318,14 @@ select length(md5((f1[1]).c2)) from dest; truncate table src; drop table src; -select length(md5((f1[1]).c2)) from dest; +select length(sha256((f1[1]).c2)) from dest; length -------- 32 (1 row) drop table dest; -drop type textandtext; +drop type byteaandbytea; -- Tests for polymorphic-array form of width_bucket() -- this exercises the varwidth and float8 code paths SELECT diff --git a/src/test/regress/expected/brin.out b/src/test/regress/expected/brin.out index 73fa38396e..d4a139e50b 100644 --- a/src/test/regress/expected/brin.out +++ b/src/test/regress/expected/brin.out @@ -530,7 +530,7 @@ EXPLAIN (COSTS OFF) SELECT * FROM brin_test WHERE b = 1; CREATE TABLE brintest_3 (a text, b text, c text, d text); -- long random strings (~2000 chars each, so ~6kB for min/max on two -- columns) to trigger toasting -WITH rand_value AS (SELECT string_agg(md5(i::text),'') AS val FROM generate_series(1,60) s(i)) +WITH rand_value AS (SELECT string_agg(encode(sha256(i::text::bytea),'hex'),'') AS val FROM generate_series(1,30) s(i)) INSERT INTO brintest_3 SELECT val, val, val, val FROM rand_value; CREATE INDEX brin_test_toast_idx ON brintest_3 USING brin (b, c); @@ -545,7 +545,7 @@ VACUUM brintest_3; -- retry insert with a different random-looking (but deterministic) value -- the value is different, and so should replace either min or max in the -- brin summary -WITH rand_value AS (SELECT string_agg(md5((-i)::text),'') AS val FROM generate_series(1,60) s(i)) +WITH rand_value AS (SELECT string_agg(encode(sha256((-i)::text::bytea),'hex'),'') AS val FROM generate_series(1,30) s(i)) INSERT INTO brintest_3 SELECT val, val, val, val FROM rand_value; -- now try some queries, accessing the brin index diff --git a/src/test/regress/expected/brin_multi.out b/src/test/regress/expected/brin_multi.out index f3309f433f..512629a257 100644 --- a/src/test/regress/expected/brin_multi.out +++ b/src/test/regress/expected/brin_multi.out @@ -29,7 +29,7 @@ INSERT INTO brintest_multi SELECT (four + 1.0)/(hundred+1), odd::float8 / (tenthous + 1), format('%s:00:%s:00:%s:00', to_hex(odd), to_hex(even), to_hex(hundred))::macaddr, - substr(md5(unique1::text), 1, 16)::macaddr8, + substr(encode(sha256(unique1::text::bytea), 'hex'), 1, 16)::macaddr8, inet '10.2.3.4/24' + tenthous, cidr '10.2.3/24' + tenthous, date '1995-08-15' + tenthous, @@ -178,8 +178,8 @@ INSERT INTO brinopers_multi VALUES '{99, 100, 2, 100, 100}'), ('macaddr8col', 'macaddr8', '{>, >=, =, <=, <}', - '{b1:d1:0e:7b:af:a4:42:12, d9:35:91:bd:f7:86:0e:1e, 72:8f:20:6c:2a:01:bf:57, 23:e8:46:63:86:07:ad:cb, 13:16:8e:6a:2e:6c:84:b4}', - '{33, 15, 1, 13, 6}'), + '{83:f8:14:f7:a9:2e:36:5c, fe:25:92:b4:2a:72:7e:97, 67:93:f9:e2:66:ce:6e:bd, 80:5c:94:a3:58:c1:d4:59, fd:24:28:59:bc:18:ff:bb}', + '{47, 2, 1, 50, 97}'), ('inetcol', 'inet', '{=, <, <=, >, >=}', '{10.2.14.231/24, 255.255.255.255, 255.255.255.255, 0.0.0.0, 0.0.0.0}', @@ -327,7 +327,7 @@ INSERT INTO brintest_multi SELECT (four + 1.0)/(hundred+1), odd::float8 / (tenthous + 1), format('%s:00:%s:00:%s:00', to_hex(odd), to_hex(even), to_hex(hundred))::macaddr, - substr(md5(unique1::text), 1, 16)::macaddr8, + substr(encode(sha256(unique1::text::bytea), 'hex'), 1, 16)::macaddr8, inet '10.2.3.4' + tenthous, cidr '10.2.3/24' + tenthous, date '1995-08-15' + tenthous, diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out index e06ac93a36..ec6dfe2633 100644 --- a/src/test/regress/expected/compression.out +++ b/src/test/regress/expected/compression.out @@ -102,7 +102,7 @@ SELECT pg_column_compression(f1) FROM cmmove2; -- test externally stored compressed data CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS -'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +$$ select string_agg(encode(sha256(g::text::bytea), 'hex'), '') from generate_series(1, 132) g $$; CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); SELECT pg_column_compression(f1) FROM cmdata2; @@ -123,13 +123,13 @@ SELECT SUBSTR(f1, 200, 5) FROM cmdata1; substr -------- 01234 - 8f14e + 7d4dd (2 rows) SELECT SUBSTR(f1, 200, 5) FROM cmdata2; substr -------- - 8f14e + 7d4dd (1 row) DROP TABLE cmdata2; @@ -315,10 +315,9 @@ SELECT pg_column_compression(f1) FROM cmdata; -- test expression index DROP TABLE cmdata2; -CREATE TABLE cmdata2 (f1 TEXT COMPRESSION pglz, f2 TEXT COMPRESSION lz4); +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz, f2 text COMPRESSION lz4); CREATE UNIQUE INDEX idx1 ON cmdata2 ((f1 || f2)); -INSERT INTO cmdata2 VALUES((SELECT array_agg(md5(g::TEXT))::TEXT FROM -generate_series(1, 50) g), VERSION()); +INSERT INTO cmdata2 VALUES ((SELECT string_agg(encode(sha256(g::text::bytea), 'hex'), '') FROM generate_series(1, 13) g), version()); -- check data is ok SELECT length(f1) FROM cmdata; length @@ -331,7 +330,7 @@ SELECT length(f1) FROM cmdata1; length -------- 10040 - 12449 + 12448 (2 rows) SELECT length(f1) FROM cmmove1; diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out index c0a47646eb..6047c7e0ed 100644 --- a/src/test/regress/expected/compression_1.out +++ b/src/test/regress/expected/compression_1.out @@ -102,7 +102,7 @@ SELECT pg_column_compression(f1) FROM cmmove2; -- test externally stored compressed data CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS -'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +$$ select string_agg(encode(sha256(g::text::bytea), 'hex'), '') from generate_series(1, 132) g $$; CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); SELECT pg_column_compression(f1) FROM cmdata2; @@ -126,7 +126,7 @@ LINE 1: SELECT SUBSTR(f1, 200, 5) FROM cmdata1; SELECT SUBSTR(f1, 200, 5) FROM cmdata2; substr -------- - 8f14e + 7d4dd (1 row) DROP TABLE cmdata2; @@ -307,15 +307,14 @@ SELECT pg_column_compression(f1) FROM cmdata; -- test expression index DROP TABLE cmdata2; -CREATE TABLE cmdata2 (f1 TEXT COMPRESSION pglz, f2 TEXT COMPRESSION lz4); +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz, f2 text COMPRESSION lz4); ERROR: compression method lz4 not supported DETAIL: This functionality requires the server to be built with lz4 support. CREATE UNIQUE INDEX idx1 ON cmdata2 ((f1 || f2)); ERROR: relation "cmdata2" does not exist -INSERT INTO cmdata2 VALUES((SELECT array_agg(md5(g::TEXT))::TEXT FROM -generate_series(1, 50) g), VERSION()); +INSERT INTO cmdata2 VALUES ((SELECT string_agg(encode(sha256(g::text::bytea), 'hex'), '') FROM generate_series(1, 13) g), version()); ERROR: relation "cmdata2" does not exist -LINE 1: INSERT INTO cmdata2 VALUES((SELECT array_agg(md5(g::TEXT))::... +LINE 1: INSERT INTO cmdata2 VALUES ((SELECT string_agg(encode(sha256... ^ -- check data is ok SELECT length(f1) FROM cmdata; diff --git a/src/test/regress/expected/inherit.out b/src/test/regress/expected/inherit.out index 2d49e765de..18509202a9 100644 --- a/src/test/regress/expected/inherit.out +++ b/src/test/regress/expected/inherit.out @@ -2450,7 +2450,7 @@ alter table permtest_child attach partition permtest_grandchild for values in (' alter table permtest_parent attach partition permtest_child for values in (1); create index on permtest_parent (left(c, 3)); insert into permtest_parent - select 1, 'a', left(md5(i::text), 5) from generate_series(0, 100) i; + select 1, 'a', left(encode(sha256(i::text::bytea), 'hex'), 5) from generate_series(0, 100) i; analyze permtest_parent; create role regress_no_child_access; revoke all on permtest_grandchild from regress_no_child_access; diff --git a/src/test/regress/expected/largeobject.out b/src/test/regress/expected/largeobject.out index 31fba2ff9d..5e7b08244c 100644 --- a/src/test/regress/expected/largeobject.out +++ b/src/test/regress/expected/largeobject.out @@ -441,7 +441,7 @@ TRUNCATE lotest_stash_values; \set newloid_1 :LASTOID SELECT lo_from_bytea(0, lo_get(:newloid_1)) AS newloid_2 \gset -SELECT md5(lo_get(:newloid_1)) = md5(lo_get(:newloid_2)); +SELECT sha256(lo_get(:newloid_1)) = sha256(lo_get(:newloid_2)); ?column? ---------- t diff --git a/src/test/regress/expected/largeobject_1.out b/src/test/regress/expected/largeobject_1.out index 7acd7f73e1..46d7388e98 100644 --- a/src/test/regress/expected/largeobject_1.out +++ b/src/test/regress/expected/largeobject_1.out @@ -441,7 +441,7 @@ TRUNCATE lotest_stash_values; \set newloid_1 :LASTOID SELECT lo_from_bytea(0, lo_get(:newloid_1)) AS newloid_2 \gset -SELECT md5(lo_get(:newloid_1)) = md5(lo_get(:newloid_2)); +SELECT sha256(lo_get(:newloid_1)) = sha256(lo_get(:newloid_2)); ?column? ---------- t diff --git a/src/test/regress/expected/matview.out b/src/test/regress/expected/matview.out index 87b6e569a5..fc489a8508 100644 --- a/src/test/regress/expected/matview.out +++ b/src/test/regress/expected/matview.out @@ -556,10 +556,10 @@ SET ROLE regress_user_mvtest; -- duplicate all the aliases used in those queries CREATE TABLE mvtest_foo_data AS SELECT i, i+1 AS tid, - md5(random()::text) AS mv, - md5(random()::text) AS newdata, - md5(random()::text) AS newdata2, - md5(random()::text) AS diff + sha256(random()::text::bytea) AS mv, + sha256(random()::text::bytea) AS newdata, + sha256(random()::text::bytea) AS newdata2, + sha256(random()::text::bytea) AS diff FROM generate_series(1, 10) i; CREATE MATERIALIZED VIEW mvtest_mv_foo AS SELECT * FROM mvtest_foo_data; CREATE MATERIALIZED VIEW mvtest_mv_foo AS SELECT * FROM mvtest_foo_data; diff --git a/src/test/regress/expected/memoize.out b/src/test/regress/expected/memoize.out index 60cbdeec7a..ecdc646985 100644 --- a/src/test/regress/expected/memoize.out +++ b/src/test/regress/expected/memoize.out @@ -164,7 +164,7 @@ DROP TABLE flt; CREATE TABLE strtest (n name, t text); CREATE INDEX strtest_n_idx ON strtest (n); CREATE INDEX strtest_t_idx ON strtest (t); -INSERT INTO strtest VALUES('one','one'),('two','two'),('three',repeat(md5('three'),100)); +INSERT INTO strtest VALUES('one','one'),('two','two'),('three',repeat(encode(sha256('three'),'hex'),50)); -- duplicate rows so we get some cache hits INSERT INTO strtest SELECT * FROM strtest; ANALYZE strtest; diff --git a/src/test/regress/expected/plpgsql.out b/src/test/regress/expected/plpgsql.out index cdc519256a..3dfbfeb7c9 100644 --- a/src/test/regress/expected/plpgsql.out +++ b/src/test/regress/expected/plpgsql.out @@ -3404,22 +3404,22 @@ select * from ret_query1(); create type record_type as (x text, y int, z boolean); create or replace function ret_query2(lim int) returns setof record_type as $$ begin - return query select md5(s.x::text), s.x, s.x > 0 + return query select encode(sha256(s.x::text::bytea), 'hex'), s.x, s.x > 0 from generate_series(-8, lim) s (x) where s.x % 2 = 0; end; $$ language plpgsql; select * from ret_query2(8); - x | y | z -----------------------------------+----+--- - a8d2ec85eaf98407310b72eb73dda247 | -8 | f - 596a3d04481816330f07e4f97510c28f | -6 | f - 0267aaf632e87a63288a08331f22c7c3 | -4 | f - 5d7b9adcbe1c629ec722529dd12e5129 | -2 | f - cfcd208495d565ef66e7dff9f98764da | 0 | f - c81e728d9d4c2f636f067f89cc14862c | 2 | t - a87ff679a2f3e71d9181a67b7542122c | 4 | t - 1679091c5a880faf6fb5e6087eb1b2dc | 6 | t - c9f0f895fb98ab9159f51fd0297e236d | 8 | t + x | y | z +------------------------------------------------------------------+----+--- + e91592205d3881e3ea35d66973bb4898d7282126ab5afcb355c95d2fc17f3822 | -8 | f + 03b26944890929ff751653acb2f2af795cee38f937f379f52ed654a68ce91216 | -6 | f + e5e0093f285a4fb94c3fcc2ad7fd04edd10d429ccda87a9aa5e4718efadf182e | -4 | f + cf3bae39dd692048a8bf961182e6a34dfd323eeb0748e162eaf055107f1cb873 | -2 | f + 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 | 0 | f + d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 | 2 | t + 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a | 4 | t + e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 | 6 | t + 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 | 8 | t (9 rows) -- test EXECUTE USING diff --git a/src/test/regress/expected/rowsecurity.out b/src/test/regress/expected/rowsecurity.out index a415ad168c..7a237ee51b 100644 --- a/src/test/regress/expected/rowsecurity.out +++ b/src/test/regress/expected/rowsecurity.out @@ -1408,9 +1408,9 @@ ERROR: infinite recursion detected in policy for relation "rec1" -- SET SESSION AUTHORIZATION regress_rls_alice; CREATE TABLE s1 (a int, b text); -INSERT INTO s1 (SELECT x, md5(x::text) FROM generate_series(-10,10) x); +INSERT INTO s1 (SELECT x, encode(sha256(x::text::bytea), 'hex') FROM generate_series(-10,10) x); CREATE TABLE s2 (x int, y text); -INSERT INTO s2 (SELECT x, md5(x::text) FROM generate_series(-6,6) x); +INSERT INTO s2 (SELECT x, encode(sha256(x::text::bytea), 'hex') FROM generate_series(-6,6) x); GRANT SELECT ON s1, s2 TO regress_rls_bob; CREATE POLICY p1 ON s1 USING (a in (select x from s2 where y like '%2f%')); CREATE POLICY p2 ON s2 USING (x in (select a from s1 where b like '%22%')); @@ -1428,13 +1428,11 @@ DROP POLICY p3 on s1; ALTER POLICY p2 ON s2 USING (x % 2 = 0); SET SESSION AUTHORIZATION regress_rls_bob; SELECT * FROM s1 WHERE f_leak(b); -- OK -NOTICE: f_leak => c81e728d9d4c2f636f067f89cc14862c -NOTICE: f_leak => a87ff679a2f3e71d9181a67b7542122c - a | b ----+---------------------------------- - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c -(2 rows) +NOTICE: f_leak => 03b26944890929ff751653acb2f2af795cee38f937f379f52ed654a68ce91216 + a | b +----+------------------------------------------------------------------ + -6 | 03b26944890929ff751653acb2f2af795cee38f937f379f52ed654a68ce91216 +(1 row) EXPLAIN (COSTS OFF) SELECT * FROM only s1 WHERE f_leak(b); QUERY PLAN @@ -1450,12 +1448,12 @@ SET SESSION AUTHORIZATION regress_rls_alice; ALTER POLICY p1 ON s1 USING (a in (select x from v2)); -- using VIEW in RLS policy SET SESSION AUTHORIZATION regress_rls_bob; SELECT * FROM s1 WHERE f_leak(b); -- OK -NOTICE: f_leak => 0267aaf632e87a63288a08331f22c7c3 -NOTICE: f_leak => 1679091c5a880faf6fb5e6087eb1b2dc - a | b -----+---------------------------------- - -4 | 0267aaf632e87a63288a08331f22c7c3 - 6 | 1679091c5a880faf6fb5e6087eb1b2dc +NOTICE: f_leak => 03b26944890929ff751653acb2f2af795cee38f937f379f52ed654a68ce91216 +NOTICE: f_leak => cf3bae39dd692048a8bf961182e6a34dfd323eeb0748e162eaf055107f1cb873 + a | b +----+------------------------------------------------------------------ + -6 | 03b26944890929ff751653acb2f2af795cee38f937f379f52ed654a68ce91216 + -2 | cf3bae39dd692048a8bf961182e6a34dfd323eeb0748e162eaf055107f1cb873 (2 rows) EXPLAIN (COSTS OFF) SELECT * FROM s1 WHERE f_leak(b); @@ -1469,12 +1467,11 @@ EXPLAIN (COSTS OFF) SELECT * FROM s1 WHERE f_leak(b); (5 rows) SELECT (SELECT x FROM s1 LIMIT 1) xx, * FROM s2 WHERE y like '%28%'; - xx | x | y -----+----+---------------------------------- - -6 | -6 | 596a3d04481816330f07e4f97510c28f - -4 | -4 | 0267aaf632e87a63288a08331f22c7c3 - 2 | 2 | c81e728d9d4c2f636f067f89cc14862c -(3 rows) + xx | x | y +----+----+------------------------------------------------------------------ + -4 | -4 | e5e0093f285a4fb94c3fcc2ad7fd04edd10d429ccda87a9aa5e4718efadf182e + 4 | 4 | 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a +(2 rows) EXPLAIN (COSTS OFF) SELECT (SELECT x FROM s1 LIMIT 1) xx, * FROM s2 WHERE y like '%28%'; QUERY PLAN @@ -1900,7 +1897,7 @@ NOTICE: f_leak => yyyyyy -- SET SESSION AUTHORIZATION regress_rls_alice; CREATE TABLE b1 (a int, b text); -INSERT INTO b1 (SELECT x, md5(x::text) FROM generate_series(-10,10) x); +INSERT INTO b1 (SELECT x, encode(sha256(x::text::bytea), 'hex') FROM generate_series(-10,10) x); CREATE POLICY p1 ON b1 USING (a % 2 = 0); ALTER TABLE b1 ENABLE ROW LEVEL SECURITY; GRANT ALL ON b1 TO regress_rls_bob; @@ -1918,18 +1915,18 @@ EXPLAIN (COSTS OFF) SELECT * FROM bv1 WHERE f_leak(b); (4 rows) SELECT * FROM bv1 WHERE f_leak(b); -NOTICE: f_leak => c81e728d9d4c2f636f067f89cc14862c -NOTICE: f_leak => a87ff679a2f3e71d9181a67b7542122c -NOTICE: f_leak => 1679091c5a880faf6fb5e6087eb1b2dc -NOTICE: f_leak => c9f0f895fb98ab9159f51fd0297e236d -NOTICE: f_leak => d3d9446802a44259755d38e6d163e820 - a | b -----+---------------------------------- - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 10 | d3d9446802a44259755d38e6d163e820 +NOTICE: f_leak => d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 +NOTICE: f_leak => 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a +NOTICE: f_leak => e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 +NOTICE: f_leak => 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 +NOTICE: f_leak => 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 + a | b +----+------------------------------------------------------------------ + 2 | d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 + 4 | 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a + 6 | e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 + 8 | 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 + 10 | 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 (5 rows) INSERT INTO bv1 VALUES (-1, 'xxx'); -- should fail view WCO @@ -1946,7 +1943,7 @@ EXPLAIN (COSTS OFF) UPDATE bv1 SET b = 'yyy' WHERE a = 4 AND f_leak(b); (3 rows) UPDATE bv1 SET b = 'yyy' WHERE a = 4 AND f_leak(b); -NOTICE: f_leak => a87ff679a2f3e71d9181a67b7542122c +NOTICE: f_leak => 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a EXPLAIN (COSTS OFF) DELETE FROM bv1 WHERE a = 6 AND f_leak(b); QUERY PLAN ----------------------------------------------------------------------- @@ -1956,30 +1953,30 @@ EXPLAIN (COSTS OFF) DELETE FROM bv1 WHERE a = 6 AND f_leak(b); (3 rows) DELETE FROM bv1 WHERE a = 6 AND f_leak(b); -NOTICE: f_leak => 1679091c5a880faf6fb5e6087eb1b2dc +NOTICE: f_leak => e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 SET SESSION AUTHORIZATION regress_rls_alice; SELECT * FROM b1; - a | b ------+---------------------------------- - -10 | 1b0fd9efa5279c4203b7c70233f86dbf - -9 | 252e691406782824eec43d7eadc3d256 - -8 | a8d2ec85eaf98407310b72eb73dda247 - -7 | 74687a12d3915d3c4d83f1af7b3683d5 - -6 | 596a3d04481816330f07e4f97510c28f - -5 | 47c1b025fa18ea96c33fbb6718688c0f - -4 | 0267aaf632e87a63288a08331f22c7c3 - -3 | b3149ecea4628efd23d2f86e5a723472 - -2 | 5d7b9adcbe1c629ec722529dd12e5129 - -1 | 6bb61e3b7bce0931da574d19d1d82c88 - 0 | cfcd208495d565ef66e7dff9f98764da - 1 | c4ca4238a0b923820dcc509a6f75849b - 2 | c81e728d9d4c2f636f067f89cc14862c - 3 | eccbc87e4b5ce2fe28308fd9f2a7baf3 - 5 | e4da3b7fbbce2345d7772b0674a318d5 - 7 | 8f14e45fceea167a5a36dedd4bea2543 - 8 | c9f0f895fb98ab9159f51fd0297e236d - 9 | 45c48cce2e2d7fbdea1afc51c7c6ad26 - 10 | d3d9446802a44259755d38e6d163e820 + a | b +-----+------------------------------------------------------------------ + -10 | c171d4ec282b23db89a99880cd624e9ba2940c1d894783602edab5d7481dc1ea + -9 | d5c534fde62beb89c745a59952c8efed8b7523cbd047e682782e4367de9ea3bf + -8 | e91592205d3881e3ea35d66973bb4898d7282126ab5afcb355c95d2fc17f3822 + -7 | a770d3270c9dcdedf12ed9fd70444f7c8a95c26cae3cae9bd867499090a2f14b + -6 | 03b26944890929ff751653acb2f2af795cee38f937f379f52ed654a68ce91216 + -5 | 37aa1ccf80e481832b2db282d4d4f895ee1e31219b7d0f6aee8dc8968828341b + -4 | e5e0093f285a4fb94c3fcc2ad7fd04edd10d429ccda87a9aa5e4718efadf182e + -3 | 615bdd17c2556f82f384392ea8557f8cc88b03501c759e23093ab0b2a9b5cd48 + -2 | cf3bae39dd692048a8bf961182e6a34dfd323eeb0748e162eaf055107f1cb873 + -1 | 1bad6b8cf97131fceab8543e81f7757195fbb1d36b376ee994ad1cf17699c464 + 0 | 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 + 1 | 6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b + 2 | d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 + 3 | 4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce + 5 | ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d + 7 | 7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451 + 8 | 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 + 9 | 19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7 + 10 | 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 12 | xxx 4 | yyy (21 rows) @@ -3038,41 +3035,41 @@ DROP VIEW rls_sbv; -- Expression structure -- SET SESSION AUTHORIZATION regress_rls_alice; -INSERT INTO y2 (SELECT x, md5(x::text) FROM generate_series(0,20) x); +INSERT INTO y2 (SELECT x, encode(sha256(x::text::bytea), 'hex') FROM generate_series(0,20) x); CREATE POLICY p2 ON y2 USING (a % 3 = 0); CREATE POLICY p3 ON y2 USING (a % 4 = 0); SET SESSION AUTHORIZATION regress_rls_bob; SELECT * FROM y2 WHERE f_leak(b); -NOTICE: f_leak => cfcd208495d565ef66e7dff9f98764da -NOTICE: f_leak => c81e728d9d4c2f636f067f89cc14862c -NOTICE: f_leak => eccbc87e4b5ce2fe28308fd9f2a7baf3 -NOTICE: f_leak => a87ff679a2f3e71d9181a67b7542122c -NOTICE: f_leak => 1679091c5a880faf6fb5e6087eb1b2dc -NOTICE: f_leak => c9f0f895fb98ab9159f51fd0297e236d -NOTICE: f_leak => 45c48cce2e2d7fbdea1afc51c7c6ad26 -NOTICE: f_leak => d3d9446802a44259755d38e6d163e820 -NOTICE: f_leak => c20ad4d76fe97759aa27a0c99bff6710 -NOTICE: f_leak => aab3238922bcc25a6f606eb525ffdc56 -NOTICE: f_leak => 9bf31c7ff062936a96d3c8bd1f8f2ff3 -NOTICE: f_leak => c74d97b01eae257e44aa9d5bade97baf -NOTICE: f_leak => 6f4922f45568161a8cdf4ad2299f6d23 -NOTICE: f_leak => 98f13708210194c475687be6106a3b84 - a | b -----+---------------------------------- - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 3 | eccbc87e4b5ce2fe28308fd9f2a7baf3 - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 9 | 45c48cce2e2d7fbdea1afc51c7c6ad26 - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 15 | 9bf31c7ff062936a96d3c8bd1f8f2ff3 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 +NOTICE: f_leak => 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 +NOTICE: f_leak => d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 +NOTICE: f_leak => 4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce +NOTICE: f_leak => 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a +NOTICE: f_leak => e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 +NOTICE: f_leak => 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 +NOTICE: f_leak => 19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7 +NOTICE: f_leak => 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 +NOTICE: f_leak => 6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918 +NOTICE: f_leak => 8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61 +NOTICE: f_leak => e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb +NOTICE: f_leak => b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9 +NOTICE: f_leak => 4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a +NOTICE: f_leak => f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b + a | b +----+------------------------------------------------------------------ + 0 | 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 + 2 | d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 + 3 | 4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce + 4 | 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a + 6 | e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 + 8 | 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 + 9 | 19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7 + 10 | 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 + 12 | 6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918 + 14 | 8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61 + 15 | e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb + 16 | b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9 + 18 | 4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a + 20 | f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b (14 rows) EXPLAIN (COSTS OFF) SELECT * FROM y2 WHERE f_leak(b); @@ -3107,22 +3104,22 @@ NOTICE: f_leak => abc NOTICE: f_leak => abc NOTICE: f_leak => abc NOTICE: f_leak => abc - a | b -----+---------------------------------- - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 3 | eccbc87e4b5ce2fe28308fd9f2a7baf3 - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 9 | 45c48cce2e2d7fbdea1afc51c7c6ad26 - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 15 | 9bf31c7ff062936a96d3c8bd1f8f2ff3 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 + a | b +----+------------------------------------------------------------------ + 0 | 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 + 2 | d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 + 3 | 4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce + 4 | 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a + 6 | e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 + 8 | 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 + 9 | 19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7 + 10 | 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 + 12 | 6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918 + 14 | 8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61 + 15 | e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb + 16 | b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9 + 18 | 4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a + 20 | f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b (14 rows) EXPLAIN (COSTS OFF) SELECT * FROM y2 WHERE f_leak('abc'); @@ -3156,20 +3153,20 @@ EXPLAIN (COSTS OFF) SELECT * FROM y2 JOIN test_qual_pushdown ON (b = abc) WHERE (7 rows) SELECT * FROM y2 JOIN test_qual_pushdown ON (b = abc) WHERE f_leak(b); -NOTICE: f_leak => cfcd208495d565ef66e7dff9f98764da -NOTICE: f_leak => c81e728d9d4c2f636f067f89cc14862c -NOTICE: f_leak => eccbc87e4b5ce2fe28308fd9f2a7baf3 -NOTICE: f_leak => a87ff679a2f3e71d9181a67b7542122c -NOTICE: f_leak => 1679091c5a880faf6fb5e6087eb1b2dc -NOTICE: f_leak => c9f0f895fb98ab9159f51fd0297e236d -NOTICE: f_leak => 45c48cce2e2d7fbdea1afc51c7c6ad26 -NOTICE: f_leak => d3d9446802a44259755d38e6d163e820 -NOTICE: f_leak => c20ad4d76fe97759aa27a0c99bff6710 -NOTICE: f_leak => aab3238922bcc25a6f606eb525ffdc56 -NOTICE: f_leak => 9bf31c7ff062936a96d3c8bd1f8f2ff3 -NOTICE: f_leak => c74d97b01eae257e44aa9d5bade97baf -NOTICE: f_leak => 6f4922f45568161a8cdf4ad2299f6d23 -NOTICE: f_leak => 98f13708210194c475687be6106a3b84 +NOTICE: f_leak => 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 +NOTICE: f_leak => d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 +NOTICE: f_leak => 4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce +NOTICE: f_leak => 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a +NOTICE: f_leak => e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 +NOTICE: f_leak => 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 +NOTICE: f_leak => 19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7 +NOTICE: f_leak => 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 +NOTICE: f_leak => 6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918 +NOTICE: f_leak => 8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61 +NOTICE: f_leak => e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb +NOTICE: f_leak => b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9 +NOTICE: f_leak => 4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a +NOTICE: f_leak => f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b a | b | abc ---+---+----- (0 rows) @@ -3239,33 +3236,33 @@ CREATE TABLE t1 (a integer, b text); CREATE POLICY p1 ON t1 USING (a % 2 = 0); ALTER TABLE t1 ENABLE ROW LEVEL SECURITY; GRANT ALL ON t1 TO regress_rls_bob; -INSERT INTO t1 (SELECT x, md5(x::text) FROM generate_series(0,20) x); +INSERT INTO t1 (SELECT x, encode(sha256(x::text::bytea), 'hex') FROM generate_series(0,20) x); SET SESSION AUTHORIZATION regress_rls_bob; WITH cte1 AS MATERIALIZED (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1; -NOTICE: f_leak => cfcd208495d565ef66e7dff9f98764da -NOTICE: f_leak => c81e728d9d4c2f636f067f89cc14862c -NOTICE: f_leak => a87ff679a2f3e71d9181a67b7542122c -NOTICE: f_leak => 1679091c5a880faf6fb5e6087eb1b2dc -NOTICE: f_leak => c9f0f895fb98ab9159f51fd0297e236d -NOTICE: f_leak => d3d9446802a44259755d38e6d163e820 -NOTICE: f_leak => c20ad4d76fe97759aa27a0c99bff6710 -NOTICE: f_leak => aab3238922bcc25a6f606eb525ffdc56 -NOTICE: f_leak => c74d97b01eae257e44aa9d5bade97baf -NOTICE: f_leak => 6f4922f45568161a8cdf4ad2299f6d23 -NOTICE: f_leak => 98f13708210194c475687be6106a3b84 - a | b -----+---------------------------------- - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 +NOTICE: f_leak => 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 +NOTICE: f_leak => d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 +NOTICE: f_leak => 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a +NOTICE: f_leak => e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 +NOTICE: f_leak => 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 +NOTICE: f_leak => 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 +NOTICE: f_leak => 6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918 +NOTICE: f_leak => 8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61 +NOTICE: f_leak => b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9 +NOTICE: f_leak => 4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a +NOTICE: f_leak => f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b + a | b +----+------------------------------------------------------------------ + 0 | 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 + 2 | d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 + 4 | 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a + 6 | e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 + 8 | 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 + 10 | 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 + 12 | 6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918 + 14 | 8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61 + 16 | b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9 + 18 | 4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a + 20 | f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b (11 rows) EXPLAIN (COSTS OFF) @@ -3281,19 +3278,19 @@ WITH cte1 AS MATERIALIZED (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1; WITH cte1 AS (UPDATE t1 SET a = a + 1 RETURNING *) SELECT * FROM cte1; --fail ERROR: new row violates row-level security policy for table "t1" WITH cte1 AS (UPDATE t1 SET a = a RETURNING *) SELECT * FROM cte1; --ok - a | b -----+---------------------------------- - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 + a | b +----+------------------------------------------------------------------ + 0 | 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 + 2 | d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 + 4 | 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a + 6 | e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 + 8 | 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 + 10 | 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 + 12 | 6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918 + 14 | 8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61 + 16 | b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9 + 18 | 4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a + 20 | f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b (11 rows) WITH cte1 AS (INSERT INTO t1 VALUES (21, 'Fail') RETURNING *) SELECT * FROM cte1; --fail @@ -3344,19 +3341,19 @@ EXPLAIN (COSTS OFF) INSERT INTO t2 (SELECT * FROM t1); (3 rows) SELECT * FROM t2; - a | b -----+---------------------------------- - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 + a | b +----+------------------------------------------------------------------ + 0 | 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 + 2 | d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 + 4 | 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a + 6 | e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 + 8 | 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 + 10 | 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 + 12 | 6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918 + 14 | 8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61 + 16 | b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9 + 18 | 4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a + 20 | f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b 20 | Success (12 rows) @@ -3368,37 +3365,37 @@ EXPLAIN (COSTS OFF) SELECT * FROM t2; CREATE TABLE t3 AS SELECT * FROM t1; SELECT * FROM t3; - a | b -----+---------------------------------- - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 + a | b +----+------------------------------------------------------------------ + 0 | 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 + 2 | d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 + 4 | 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a + 6 | e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 + 8 | 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 + 10 | 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 + 12 | 6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918 + 14 | 8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61 + 16 | b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9 + 18 | 4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a + 20 | f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b 20 | Success (12 rows) SELECT * INTO t4 FROM t1; SELECT * FROM t4; - a | b -----+---------------------------------- - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 + a | b +----+------------------------------------------------------------------ + 0 | 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 + 2 | d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 + 4 | 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a + 6 | e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 + 8 | 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 + 10 | 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 + 12 | 6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918 + 14 | 8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61 + 16 | b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9 + 18 | 4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a + 20 | f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b 20 | Success (12 rows) @@ -3469,29 +3466,29 @@ ALTER TABLE t1 OWNER TO regress_rls_alice; -- Check that default deny does not apply to superuser. RESET SESSION AUTHORIZATION; SELECT * FROM t1; - a | b -----+---------------------------------- - 1 | c4ca4238a0b923820dcc509a6f75849b - 3 | eccbc87e4b5ce2fe28308fd9f2a7baf3 - 5 | e4da3b7fbbce2345d7772b0674a318d5 - 7 | 8f14e45fceea167a5a36dedd4bea2543 - 9 | 45c48cce2e2d7fbdea1afc51c7c6ad26 - 11 | 6512bd43d9caa6e02c990b0a82652dca - 13 | c51ce410c124a10e0db5e4b97fc2af39 - 15 | 9bf31c7ff062936a96d3c8bd1f8f2ff3 - 17 | 70efdf2ec9b086079795c442636b55fb - 19 | 1f0e3dad99908345f7439f8ffabdffc4 - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 + a | b +----+------------------------------------------------------------------ + 1 | 6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b + 3 | 4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce + 5 | ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d + 7 | 7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451 + 9 | 19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7 + 11 | 4fc82b26aecb47d2868c4efbe3581732a3e7cbcc6c2efb32062c08170a05eeb8 + 13 | 3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278 + 15 | e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb + 17 | 4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3 + 19 | 9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767 + 0 | 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 + 2 | d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 + 4 | 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a + 6 | e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 + 8 | 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 + 10 | 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 + 12 | 6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918 + 14 | 8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61 + 16 | b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9 + 18 | 4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a + 20 | f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b 20 | Success (22 rows) @@ -3504,29 +3501,29 @@ EXPLAIN (COSTS OFF) SELECT * FROM t1; -- Check that default deny does not apply to table owner. SET SESSION AUTHORIZATION regress_rls_alice; SELECT * FROM t1; - a | b -----+---------------------------------- - 1 | c4ca4238a0b923820dcc509a6f75849b - 3 | eccbc87e4b5ce2fe28308fd9f2a7baf3 - 5 | e4da3b7fbbce2345d7772b0674a318d5 - 7 | 8f14e45fceea167a5a36dedd4bea2543 - 9 | 45c48cce2e2d7fbdea1afc51c7c6ad26 - 11 | 6512bd43d9caa6e02c990b0a82652dca - 13 | c51ce410c124a10e0db5e4b97fc2af39 - 15 | 9bf31c7ff062936a96d3c8bd1f8f2ff3 - 17 | 70efdf2ec9b086079795c442636b55fb - 19 | 1f0e3dad99908345f7439f8ffabdffc4 - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 + a | b +----+------------------------------------------------------------------ + 1 | 6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b + 3 | 4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce + 5 | ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d + 7 | 7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451 + 9 | 19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7 + 11 | 4fc82b26aecb47d2868c4efbe3581732a3e7cbcc6c2efb32062c08170a05eeb8 + 13 | 3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278 + 15 | e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb + 17 | 4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3 + 19 | 9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767 + 0 | 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 + 2 | d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 + 4 | 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a + 6 | e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 + 8 | 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 + 10 | 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 + 12 | 6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918 + 14 | 8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61 + 16 | b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9 + 18 | 4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a + 20 | f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b 20 | Success (22 rows) @@ -3574,35 +3571,35 @@ CREATE TABLE copy_t (a integer, b text); CREATE POLICY p1 ON copy_t USING (a % 2 = 0); ALTER TABLE copy_t ENABLE ROW LEVEL SECURITY; GRANT ALL ON copy_t TO regress_rls_bob, regress_rls_exempt_user; -INSERT INTO copy_t (SELECT x, md5(x::text) FROM generate_series(0,10) x); +INSERT INTO copy_t (SELECT x, encode(sha256(x::text::bytea), 'hex') FROM generate_series(0,10) x); -- Check COPY TO as Superuser/owner. RESET SESSION AUTHORIZATION; SET row_security TO OFF; COPY (SELECT * FROM copy_t ORDER BY a ASC) TO STDOUT WITH DELIMITER ','; -0,cfcd208495d565ef66e7dff9f98764da -1,c4ca4238a0b923820dcc509a6f75849b -2,c81e728d9d4c2f636f067f89cc14862c -3,eccbc87e4b5ce2fe28308fd9f2a7baf3 -4,a87ff679a2f3e71d9181a67b7542122c -5,e4da3b7fbbce2345d7772b0674a318d5 -6,1679091c5a880faf6fb5e6087eb1b2dc -7,8f14e45fceea167a5a36dedd4bea2543 -8,c9f0f895fb98ab9159f51fd0297e236d -9,45c48cce2e2d7fbdea1afc51c7c6ad26 -10,d3d9446802a44259755d38e6d163e820 +0,5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 +1,6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b +2,d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 +3,4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce +4,4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a +5,ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d +6,e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 +7,7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451 +8,2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 +9,19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7 +10,4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 SET row_security TO ON; COPY (SELECT * FROM copy_t ORDER BY a ASC) TO STDOUT WITH DELIMITER ','; -0,cfcd208495d565ef66e7dff9f98764da -1,c4ca4238a0b923820dcc509a6f75849b -2,c81e728d9d4c2f636f067f89cc14862c -3,eccbc87e4b5ce2fe28308fd9f2a7baf3 -4,a87ff679a2f3e71d9181a67b7542122c -5,e4da3b7fbbce2345d7772b0674a318d5 -6,1679091c5a880faf6fb5e6087eb1b2dc -7,8f14e45fceea167a5a36dedd4bea2543 -8,c9f0f895fb98ab9159f51fd0297e236d -9,45c48cce2e2d7fbdea1afc51c7c6ad26 -10,d3d9446802a44259755d38e6d163e820 +0,5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 +1,6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b +2,d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 +3,4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce +4,4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a +5,ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d +6,e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 +7,7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451 +8,2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 +9,19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7 +10,4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 -- Check COPY TO as user with permissions. SET SESSION AUTHORIZATION regress_rls_bob; SET row_security TO OFF; @@ -3610,40 +3607,40 @@ COPY (SELECT * FROM copy_t ORDER BY a ASC) TO STDOUT WITH DELIMITER ','; --fail ERROR: query would be affected by row-level security policy for table "copy_t" SET row_security TO ON; COPY (SELECT * FROM copy_t ORDER BY a ASC) TO STDOUT WITH DELIMITER ','; --ok -0,cfcd208495d565ef66e7dff9f98764da -2,c81e728d9d4c2f636f067f89cc14862c -4,a87ff679a2f3e71d9181a67b7542122c -6,1679091c5a880faf6fb5e6087eb1b2dc -8,c9f0f895fb98ab9159f51fd0297e236d -10,d3d9446802a44259755d38e6d163e820 +0,5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 +2,d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 +4,4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a +6,e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 +8,2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 +10,4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 -- Check COPY TO as user with permissions and BYPASSRLS SET SESSION AUTHORIZATION regress_rls_exempt_user; SET row_security TO OFF; COPY (SELECT * FROM copy_t ORDER BY a ASC) TO STDOUT WITH DELIMITER ','; --ok -0,cfcd208495d565ef66e7dff9f98764da -1,c4ca4238a0b923820dcc509a6f75849b -2,c81e728d9d4c2f636f067f89cc14862c -3,eccbc87e4b5ce2fe28308fd9f2a7baf3 -4,a87ff679a2f3e71d9181a67b7542122c -5,e4da3b7fbbce2345d7772b0674a318d5 -6,1679091c5a880faf6fb5e6087eb1b2dc -7,8f14e45fceea167a5a36dedd4bea2543 -8,c9f0f895fb98ab9159f51fd0297e236d -9,45c48cce2e2d7fbdea1afc51c7c6ad26 -10,d3d9446802a44259755d38e6d163e820 +0,5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 +1,6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b +2,d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 +3,4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce +4,4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a +5,ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d +6,e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 +7,7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451 +8,2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 +9,19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7 +10,4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 SET row_security TO ON; COPY (SELECT * FROM copy_t ORDER BY a ASC) TO STDOUT WITH DELIMITER ','; --ok -0,cfcd208495d565ef66e7dff9f98764da -1,c4ca4238a0b923820dcc509a6f75849b -2,c81e728d9d4c2f636f067f89cc14862c -3,eccbc87e4b5ce2fe28308fd9f2a7baf3 -4,a87ff679a2f3e71d9181a67b7542122c -5,e4da3b7fbbce2345d7772b0674a318d5 -6,1679091c5a880faf6fb5e6087eb1b2dc -7,8f14e45fceea167a5a36dedd4bea2543 -8,c9f0f895fb98ab9159f51fd0297e236d -9,45c48cce2e2d7fbdea1afc51c7c6ad26 -10,d3d9446802a44259755d38e6d163e820 +0,5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 +1,6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b +2,d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 +3,4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce +4,4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a +5,ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d +6,e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 +7,7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451 +8,2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 +9,19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7 +10,4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 -- Check COPY TO as user without permissions. SET row_security TO OFF; SET SESSION AUTHORIZATION regress_rls_carol; SET row_security TO OFF; @@ -3659,15 +3656,15 @@ CREATE TABLE copy_rel_to (a integer, b text); CREATE POLICY p1 ON copy_rel_to USING (a % 2 = 0); ALTER TABLE copy_rel_to ENABLE ROW LEVEL SECURITY; GRANT ALL ON copy_rel_to TO regress_rls_bob, regress_rls_exempt_user; -INSERT INTO copy_rel_to VALUES (1, md5('1')); +INSERT INTO copy_rel_to VALUES (1, encode(sha256('1'), 'hex')); -- Check COPY TO as Superuser/owner. RESET SESSION AUTHORIZATION; SET row_security TO OFF; COPY copy_rel_to TO STDOUT WITH DELIMITER ','; -1,c4ca4238a0b923820dcc509a6f75849b +1,6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b SET row_security TO ON; COPY copy_rel_to TO STDOUT WITH DELIMITER ','; -1,c4ca4238a0b923820dcc509a6f75849b +1,6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b -- Check COPY TO as user with permissions. SET SESSION AUTHORIZATION regress_rls_bob; SET row_security TO OFF; @@ -3679,10 +3676,10 @@ COPY copy_rel_to TO STDOUT WITH DELIMITER ','; --ok SET SESSION AUTHORIZATION regress_rls_exempt_user; SET row_security TO OFF; COPY copy_rel_to TO STDOUT WITH DELIMITER ','; --ok -1,c4ca4238a0b923820dcc509a6f75849b +1,6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b SET row_security TO ON; COPY copy_rel_to TO STDOUT WITH DELIMITER ','; --ok -1,c4ca4238a0b923820dcc509a6f75849b +1,6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b -- Check COPY TO as user without permissions. SET row_security TO OFF; SET SESSION AUTHORIZATION regress_rls_carol; SET row_security TO OFF; diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out index 03880874c1..41d2439388 100644 --- a/src/test/regress/expected/stats_ext.out +++ b/src/test/regress/expected/stats_ext.out @@ -2615,18 +2615,18 @@ CREATE TABLE mcv_lists_uuid ( WITH (autovacuum_enabled = off); INSERT INTO mcv_lists_uuid (a, b, c) SELECT - md5(mod(i,100)::text)::uuid, - md5(mod(i,50)::text)::uuid, - md5(mod(i,25)::text)::uuid + substr(encode(sha256(mod(i,100)::text::bytea), 'hex'), 1, 32)::uuid, + substr(encode(sha256(mod(i,50)::text::bytea), 'hex'), 1, 32)::uuid, + substr(encode(sha256(mod(i,25)::text::bytea), 'hex'), 1, 32)::uuid FROM generate_series(1,5000) s(i); ANALYZE mcv_lists_uuid; -SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND b = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'''); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND b = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'''); estimated | actual -----------+-------- 1 | 50 (1 row) -SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND b = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND c = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'''); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND b = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND c = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'''); estimated | actual -----------+-------- 1 | 50 @@ -2635,13 +2635,13 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''167 CREATE STATISTICS mcv_lists_uuid_stats (mcv) ON a, b, c FROM mcv_lists_uuid; ANALYZE mcv_lists_uuid; -SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND b = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'''); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND b = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'''); estimated | actual -----------+-------- 50 | 50 (1 row) -SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND b = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND c = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'''); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND b = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND c = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'''); estimated | actual -----------+-------- 50 | 50 diff --git a/src/test/regress/sql/arrays.sql b/src/test/regress/sql/arrays.sql index 38e8dd440b..5bdc00d26f 100644 --- a/src/test/regress/sql/arrays.sql +++ b/src/test/regress/sql/arrays.sql @@ -671,20 +671,20 @@ CREATE TEMP TABLE arraggtest ( f1 INT[], f2 TEXT[][], f3 FLOAT[]); -- Check that arrays of composites are safely detoasted when needed -create temp table src (f1 text); +create temp table src (f1 bytea); insert into src - select string_agg(random()::text,'') from generate_series(1,10000); -create type textandtext as (c1 text, c2 text); -create temp table dest (f1 textandtext[]); -insert into dest select array[row(f1,f1)::textandtext] from src; -select length(md5((f1[1]).c2)) from dest; + select string_agg(random()::text::bytea,'') from generate_series(1,10000); +create type byteaandbytea as (c1 bytea, c2 bytea); +create temp table dest (f1 byteaandbytea[]); +insert into dest select array[row(f1,f1)::byteaandbytea] from src; +select length(sha256((f1[1]).c2)) from dest; delete from src; -select length(md5((f1[1]).c2)) from dest; +select length(sha256((f1[1]).c2)) from dest; truncate table src; drop table src; -select length(md5((f1[1]).c2)) from dest; +select length(sha256((f1[1]).c2)) from dest; drop table dest; -drop type textandtext; +drop type byteaandbytea; -- Tests for polymorphic-array form of width_bucket() diff --git a/src/test/regress/sql/brin.sql b/src/test/regress/sql/brin.sql index e68e9e18df..7fa125315f 100644 --- a/src/test/regress/sql/brin.sql +++ b/src/test/regress/sql/brin.sql @@ -476,7 +476,7 @@ CREATE TABLE brintest_3 (a text, b text, c text, d text); -- long random strings (~2000 chars each, so ~6kB for min/max on two -- columns) to trigger toasting -WITH rand_value AS (SELECT string_agg(md5(i::text),'') AS val FROM generate_series(1,60) s(i)) +WITH rand_value AS (SELECT string_agg(encode(sha256(i::text::bytea),'hex'),'') AS val FROM generate_series(1,30) s(i)) INSERT INTO brintest_3 SELECT val, val, val, val FROM rand_value; @@ -495,7 +495,7 @@ CREATE INDEX CONCURRENTLY brin_test_temp_idx ON brintest_3(a); -- retry insert with a different random-looking (but deterministic) value -- the value is different, and so should replace either min or max in the -- brin summary -WITH rand_value AS (SELECT string_agg(md5((-i)::text),'') AS val FROM generate_series(1,60) s(i)) +WITH rand_value AS (SELECT string_agg(encode(sha256((-i)::text::bytea),'hex'),'') AS val FROM generate_series(1,30) s(i)) INSERT INTO brintest_3 SELECT val, val, val, val FROM rand_value; diff --git a/src/test/regress/sql/brin_multi.sql b/src/test/regress/sql/brin_multi.sql index 2189b6ccf4..0e26a8705e 100644 --- a/src/test/regress/sql/brin_multi.sql +++ b/src/test/regress/sql/brin_multi.sql @@ -30,7 +30,7 @@ CREATE TABLE brintest_multi ( (four + 1.0)/(hundred+1), odd::float8 / (tenthous + 1), format('%s:00:%s:00:%s:00', to_hex(odd), to_hex(even), to_hex(hundred))::macaddr, - substr(md5(unique1::text), 1, 16)::macaddr8, + substr(encode(sha256(unique1::text::bytea), 'hex'), 1, 16)::macaddr8, inet '10.2.3.4/24' + tenthous, cidr '10.2.3/24' + tenthous, date '1995-08-15' + tenthous, @@ -182,8 +182,8 @@ CREATE TABLE brinopers_multi (colname name, typ text, '{99, 100, 2, 100, 100}'), ('macaddr8col', 'macaddr8', '{>, >=, =, <=, <}', - '{b1:d1:0e:7b:af:a4:42:12, d9:35:91:bd:f7:86:0e:1e, 72:8f:20:6c:2a:01:bf:57, 23:e8:46:63:86:07:ad:cb, 13:16:8e:6a:2e:6c:84:b4}', - '{33, 15, 1, 13, 6}'), + '{83:f8:14:f7:a9:2e:36:5c, fe:25:92:b4:2a:72:7e:97, 67:93:f9:e2:66:ce:6e:bd, 80:5c:94:a3:58:c1:d4:59, fd:24:28:59:bc:18:ff:bb}', + '{47, 2, 1, 50, 97}'), ('inetcol', 'inet', '{=, <, <=, >, >=}', '{10.2.14.231/24, 255.255.255.255, 255.255.255.255, 0.0.0.0, 0.0.0.0}', @@ -334,7 +334,7 @@ CREATE TABLE brinopers_multi (colname name, typ text, (four + 1.0)/(hundred+1), odd::float8 / (tenthous + 1), format('%s:00:%s:00:%s:00', to_hex(odd), to_hex(even), to_hex(hundred))::macaddr, - substr(md5(unique1::text), 1, 16)::macaddr8, + substr(encode(sha256(unique1::text::bytea), 'hex'), 1, 16)::macaddr8, inet '10.2.3.4' + tenthous, cidr '10.2.3/24' + tenthous, date '1995-08-15' + tenthous, diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql index 86332dcc51..5a45bebad6 100644 --- a/src/test/regress/sql/compression.sql +++ b/src/test/regress/sql/compression.sql @@ -48,7 +48,7 @@ CREATE TABLE cmmove2(f1 text COMPRESSION pglz); -- test externally stored compressed data CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS -'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +$$ select string_agg(encode(sha256(g::text::bytea), 'hex'), '') from generate_series(1, 132) g $$; CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); SELECT pg_column_compression(f1) FROM cmdata2; @@ -133,10 +133,9 @@ CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); -- test expression index DROP TABLE cmdata2; -CREATE TABLE cmdata2 (f1 TEXT COMPRESSION pglz, f2 TEXT COMPRESSION lz4); +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz, f2 text COMPRESSION lz4); CREATE UNIQUE INDEX idx1 ON cmdata2 ((f1 || f2)); -INSERT INTO cmdata2 VALUES((SELECT array_agg(md5(g::TEXT))::TEXT FROM -generate_series(1, 50) g), VERSION()); +INSERT INTO cmdata2 VALUES ((SELECT string_agg(encode(sha256(g::text::bytea), 'hex'), '') FROM generate_series(1, 13) g), version()); -- check data is ok SELECT length(f1) FROM cmdata; diff --git a/src/test/regress/sql/inherit.sql b/src/test/regress/sql/inherit.sql index 195aedb5ff..6c15686d52 100644 --- a/src/test/regress/sql/inherit.sql +++ b/src/test/regress/sql/inherit.sql @@ -882,7 +882,7 @@ CREATE TABLE test_foreign_constraints_inh () INHERITS (test_foreign_constraints) alter table permtest_parent attach partition permtest_child for values in (1); create index on permtest_parent (left(c, 3)); insert into permtest_parent - select 1, 'a', left(md5(i::text), 5) from generate_series(0, 100) i; + select 1, 'a', left(encode(sha256(i::text::bytea), 'hex'), 5) from generate_series(0, 100) i; analyze permtest_parent; create role regress_no_child_access; revoke all on permtest_grandchild from regress_no_child_access; diff --git a/src/test/regress/sql/largeobject.sql b/src/test/regress/sql/largeobject.sql index 15e0dff7a3..207a7d343b 100644 --- a/src/test/regress/sql/largeobject.sql +++ b/src/test/regress/sql/largeobject.sql @@ -244,7 +244,7 @@ CREATE TABLE lotest_stash_values (loid oid, fd integer); SELECT lo_from_bytea(0, lo_get(:newloid_1)) AS newloid_2 \gset -SELECT md5(lo_get(:newloid_1)) = md5(lo_get(:newloid_2)); +SELECT sha256(lo_get(:newloid_1)) = sha256(lo_get(:newloid_2)); SELECT lo_get(:newloid_1, 0, 20); SELECT lo_get(:newloid_1, 10, 20); diff --git a/src/test/regress/sql/matview.sql b/src/test/regress/sql/matview.sql index 68b9ccfd45..09c55ec98d 100644 --- a/src/test/regress/sql/matview.sql +++ b/src/test/regress/sql/matview.sql @@ -216,10 +216,10 @@ CREATE ROLE regress_user_mvtest; -- duplicate all the aliases used in those queries CREATE TABLE mvtest_foo_data AS SELECT i, i+1 AS tid, - md5(random()::text) AS mv, - md5(random()::text) AS newdata, - md5(random()::text) AS newdata2, - md5(random()::text) AS diff + sha256(random()::text::bytea) AS mv, + sha256(random()::text::bytea) AS newdata, + sha256(random()::text::bytea) AS newdata2, + sha256(random()::text::bytea) AS diff FROM generate_series(1, 10) i; CREATE MATERIALIZED VIEW mvtest_mv_foo AS SELECT * FROM mvtest_foo_data; CREATE MATERIALIZED VIEW mvtest_mv_foo AS SELECT * FROM mvtest_foo_data; diff --git a/src/test/regress/sql/memoize.sql b/src/test/regress/sql/memoize.sql index d66acaed85..80870ee8a5 100644 --- a/src/test/regress/sql/memoize.sql +++ b/src/test/regress/sql/memoize.sql @@ -91,7 +91,7 @@ CREATE INDEX flt_f_idx ON flt (f); CREATE TABLE strtest (n name, t text); CREATE INDEX strtest_n_idx ON strtest (n); CREATE INDEX strtest_t_idx ON strtest (t); -INSERT INTO strtest VALUES('one','one'),('two','two'),('three',repeat(md5('three'),100)); +INSERT INTO strtest VALUES('one','one'),('two','two'),('three',repeat(encode(sha256('three'),'hex'),50)); -- duplicate rows so we get some cache hits INSERT INTO strtest SELECT * FROM strtest; ANALYZE strtest; diff --git a/src/test/regress/sql/plpgsql.sql b/src/test/regress/sql/plpgsql.sql index 9a53b15081..87246fb395 100644 --- a/src/test/regress/sql/plpgsql.sql +++ b/src/test/regress/sql/plpgsql.sql @@ -2877,7 +2877,7 @@ CREATE FUNCTION reraise_test() RETURNS void AS $$ create or replace function ret_query2(lim int) returns setof record_type as $$ begin - return query select md5(s.x::text), s.x, s.x > 0 + return query select encode(sha256(s.x::text::bytea), 'hex'), s.x, s.x > 0 from generate_series(-8, lim) s (x) where s.x % 2 = 0; end; $$ language plpgsql; diff --git a/src/test/regress/sql/rowsecurity.sql b/src/test/regress/sql/rowsecurity.sql index b38fa8ed8f..f2ba7b45b1 100644 --- a/src/test/regress/sql/rowsecurity.sql +++ b/src/test/regress/sql/rowsecurity.sql @@ -534,10 +534,10 @@ CREATE POLICY r2 ON rec2 USING (a = (SELECT x FROM rec1v WHERE y = b)); -- SET SESSION AUTHORIZATION regress_rls_alice; CREATE TABLE s1 (a int, b text); -INSERT INTO s1 (SELECT x, md5(x::text) FROM generate_series(-10,10) x); +INSERT INTO s1 (SELECT x, encode(sha256(x::text::bytea), 'hex') FROM generate_series(-10,10) x); CREATE TABLE s2 (x int, y text); -INSERT INTO s2 (SELECT x, md5(x::text) FROM generate_series(-6,6) x); +INSERT INTO s2 (SELECT x, encode(sha256(x::text::bytea), 'hex') FROM generate_series(-6,6) x); GRANT SELECT ON s1, s2 TO regress_rls_bob; @@ -669,7 +669,7 @@ CREATE VIEW v2 AS SELECT * FROM s2 WHERE y like '%af%'; -- SET SESSION AUTHORIZATION regress_rls_alice; CREATE TABLE b1 (a int, b text); -INSERT INTO b1 (SELECT x, md5(x::text) FROM generate_series(-10,10) x); +INSERT INTO b1 (SELECT x, encode(sha256(x::text::bytea), 'hex') FROM generate_series(-10,10) x); CREATE POLICY p1 ON b1 USING (a % 2 = 0); ALTER TABLE b1 ENABLE ROW LEVEL SECURITY; @@ -1269,7 +1269,7 @@ CREATE VIEW rls_sbv WITH (security_barrier) AS -- Expression structure -- SET SESSION AUTHORIZATION regress_rls_alice; -INSERT INTO y2 (SELECT x, md5(x::text) FROM generate_series(0,20) x); +INSERT INTO y2 (SELECT x, encode(sha256(x::text::bytea), 'hex') FROM generate_series(0,20) x); CREATE POLICY p2 ON y2 USING (a % 3 = 0); CREATE POLICY p3 ON y2 USING (a % 4 = 0); @@ -1341,7 +1341,7 @@ CREATE POLICY p1 ON t1 USING (a % 2 = 0); GRANT ALL ON t1 TO regress_rls_bob; -INSERT INTO t1 (SELECT x, md5(x::text) FROM generate_series(0,20) x); +INSERT INTO t1 (SELECT x, encode(sha256(x::text::bytea), 'hex') FROM generate_series(0,20) x); SET SESSION AUTHORIZATION regress_rls_bob; @@ -1473,7 +1473,7 @@ CREATE POLICY p1 ON copy_t USING (a % 2 = 0); GRANT ALL ON copy_t TO regress_rls_bob, regress_rls_exempt_user; -INSERT INTO copy_t (SELECT x, md5(x::text) FROM generate_series(0,10) x); +INSERT INTO copy_t (SELECT x, encode(sha256(x::text::bytea), 'hex') FROM generate_series(0,10) x); -- Check COPY TO as Superuser/owner. RESET SESSION AUTHORIZATION; @@ -1513,7 +1513,7 @@ CREATE POLICY p1 ON copy_rel_to USING (a % 2 = 0); GRANT ALL ON copy_rel_to TO regress_rls_bob, regress_rls_exempt_user; -INSERT INTO copy_rel_to VALUES (1, md5('1')); +INSERT INTO copy_rel_to VALUES (1, encode(sha256('1'), 'hex')); -- Check COPY TO as Superuser/owner. RESET SESSION AUTHORIZATION; diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql index d0d42cd013..19527ed50f 100644 --- a/src/test/regress/sql/stats_ext.sql +++ b/src/test/regress/sql/stats_ext.sql @@ -1283,25 +1283,25 @@ CREATE TABLE mcv_lists_uuid ( INSERT INTO mcv_lists_uuid (a, b, c) SELECT - md5(mod(i,100)::text)::uuid, - md5(mod(i,50)::text)::uuid, - md5(mod(i,25)::text)::uuid + substr(encode(sha256(mod(i,100)::text::bytea), 'hex'), 1, 32)::uuid, + substr(encode(sha256(mod(i,50)::text::bytea), 'hex'), 1, 32)::uuid, + substr(encode(sha256(mod(i,25)::text::bytea), 'hex'), 1, 32)::uuid FROM generate_series(1,5000) s(i); ANALYZE mcv_lists_uuid; -SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND b = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'''); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND b = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'''); -SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND b = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND c = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'''); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND b = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND c = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'''); CREATE STATISTICS mcv_lists_uuid_stats (mcv) ON a, b, c FROM mcv_lists_uuid; ANALYZE mcv_lists_uuid; -SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND b = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'''); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND b = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'''); -SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND b = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND c = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'''); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND b = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND c = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'''); DROP TABLE mcv_lists_uuid; -- 2.39.1 Attachments: [text/plain] v2-0001-Remove-incidental-md5-function-uses-from-main-reg.patch (72.3K, ../../[email protected]/2-v2-0001-Remove-incidental-md5-function-uses-from-main-reg.patch) download | inline diff: From 28a32d41bf93c682caba2bacd94bee0f389915da Mon Sep 17 00:00:00 2001 From: Peter Eisentraut <[email protected]> Date: Wed, 7 Dec 2022 13:32:26 +0100 Subject: [PATCH v2] Remove incidental md5() function uses from main regression tests Most of these calls were to generate some random data. These can be replaced by appropriately adapted sha256() calls. This will eventually allow these tests to pass in OpenSSL FIPS mode (which does not allow MD5 use). Similar work for other test suites will follow later. Discussion: https://www.postgresql.org/message-id/flat/[email protected] --- src/test/regress/expected/arrays.out | 18 +- src/test/regress/expected/brin.out | 4 +- src/test/regress/expected/brin_multi.out | 8 +- src/test/regress/expected/compression.out | 13 +- src/test/regress/expected/compression_1.out | 11 +- src/test/regress/expected/inherit.out | 2 +- src/test/regress/expected/largeobject.out | 2 +- src/test/regress/expected/largeobject_1.out | 2 +- src/test/regress/expected/matview.out | 8 +- src/test/regress/expected/memoize.out | 2 +- src/test/regress/expected/plpgsql.out | 24 +- src/test/regress/expected/rowsecurity.out | 591 ++++++++++---------- src/test/regress/expected/stats_ext.out | 14 +- src/test/regress/sql/arrays.sql | 18 +- src/test/regress/sql/brin.sql | 4 +- src/test/regress/sql/brin_multi.sql | 8 +- src/test/regress/sql/compression.sql | 7 +- src/test/regress/sql/inherit.sql | 2 +- src/test/regress/sql/largeobject.sql | 2 +- src/test/regress/sql/matview.sql | 8 +- src/test/regress/sql/memoize.sql | 2 +- src/test/regress/sql/plpgsql.sql | 2 +- src/test/regress/sql/rowsecurity.sql | 14 +- src/test/regress/sql/stats_ext.sql | 14 +- 24 files changed, 387 insertions(+), 393 deletions(-) diff --git a/src/test/regress/expected/arrays.out b/src/test/regress/expected/arrays.out index a2f9d7ed16..c6c084b088 100644 --- a/src/test/regress/expected/arrays.out +++ b/src/test/regress/expected/arrays.out @@ -2297,20 +2297,20 @@ select * from t1; (1 row) -- Check that arrays of composites are safely detoasted when needed -create temp table src (f1 text); +create temp table src (f1 bytea); insert into src - select string_agg(random()::text,'') from generate_series(1,10000); -create type textandtext as (c1 text, c2 text); -create temp table dest (f1 textandtext[]); -insert into dest select array[row(f1,f1)::textandtext] from src; -select length(md5((f1[1]).c2)) from dest; + select string_agg(random()::text::bytea,'') from generate_series(1,10000); +create type byteaandbytea as (c1 bytea, c2 bytea); +create temp table dest (f1 byteaandbytea[]); +insert into dest select array[row(f1,f1)::byteaandbytea] from src; +select length(sha256((f1[1]).c2)) from dest; length -------- 32 (1 row) delete from src; -select length(md5((f1[1]).c2)) from dest; +select length(sha256((f1[1]).c2)) from dest; length -------- 32 @@ -2318,14 +2318,14 @@ select length(md5((f1[1]).c2)) from dest; truncate table src; drop table src; -select length(md5((f1[1]).c2)) from dest; +select length(sha256((f1[1]).c2)) from dest; length -------- 32 (1 row) drop table dest; -drop type textandtext; +drop type byteaandbytea; -- Tests for polymorphic-array form of width_bucket() -- this exercises the varwidth and float8 code paths SELECT diff --git a/src/test/regress/expected/brin.out b/src/test/regress/expected/brin.out index 73fa38396e..d4a139e50b 100644 --- a/src/test/regress/expected/brin.out +++ b/src/test/regress/expected/brin.out @@ -530,7 +530,7 @@ EXPLAIN (COSTS OFF) SELECT * FROM brin_test WHERE b = 1; CREATE TABLE brintest_3 (a text, b text, c text, d text); -- long random strings (~2000 chars each, so ~6kB for min/max on two -- columns) to trigger toasting -WITH rand_value AS (SELECT string_agg(md5(i::text),'') AS val FROM generate_series(1,60) s(i)) +WITH rand_value AS (SELECT string_agg(encode(sha256(i::text::bytea),'hex'),'') AS val FROM generate_series(1,30) s(i)) INSERT INTO brintest_3 SELECT val, val, val, val FROM rand_value; CREATE INDEX brin_test_toast_idx ON brintest_3 USING brin (b, c); @@ -545,7 +545,7 @@ VACUUM brintest_3; -- retry insert with a different random-looking (but deterministic) value -- the value is different, and so should replace either min or max in the -- brin summary -WITH rand_value AS (SELECT string_agg(md5((-i)::text),'') AS val FROM generate_series(1,60) s(i)) +WITH rand_value AS (SELECT string_agg(encode(sha256((-i)::text::bytea),'hex'),'') AS val FROM generate_series(1,30) s(i)) INSERT INTO brintest_3 SELECT val, val, val, val FROM rand_value; -- now try some queries, accessing the brin index diff --git a/src/test/regress/expected/brin_multi.out b/src/test/regress/expected/brin_multi.out index f3309f433f..512629a257 100644 --- a/src/test/regress/expected/brin_multi.out +++ b/src/test/regress/expected/brin_multi.out @@ -29,7 +29,7 @@ INSERT INTO brintest_multi SELECT (four + 1.0)/(hundred+1), odd::float8 / (tenthous + 1), format('%s:00:%s:00:%s:00', to_hex(odd), to_hex(even), to_hex(hundred))::macaddr, - substr(md5(unique1::text), 1, 16)::macaddr8, + substr(encode(sha256(unique1::text::bytea), 'hex'), 1, 16)::macaddr8, inet '10.2.3.4/24' + tenthous, cidr '10.2.3/24' + tenthous, date '1995-08-15' + tenthous, @@ -178,8 +178,8 @@ INSERT INTO brinopers_multi VALUES '{99, 100, 2, 100, 100}'), ('macaddr8col', 'macaddr8', '{>, >=, =, <=, <}', - '{b1:d1:0e:7b:af:a4:42:12, d9:35:91:bd:f7:86:0e:1e, 72:8f:20:6c:2a:01:bf:57, 23:e8:46:63:86:07:ad:cb, 13:16:8e:6a:2e:6c:84:b4}', - '{33, 15, 1, 13, 6}'), + '{83:f8:14:f7:a9:2e:36:5c, fe:25:92:b4:2a:72:7e:97, 67:93:f9:e2:66:ce:6e:bd, 80:5c:94:a3:58:c1:d4:59, fd:24:28:59:bc:18:ff:bb}', + '{47, 2, 1, 50, 97}'), ('inetcol', 'inet', '{=, <, <=, >, >=}', '{10.2.14.231/24, 255.255.255.255, 255.255.255.255, 0.0.0.0, 0.0.0.0}', @@ -327,7 +327,7 @@ INSERT INTO brintest_multi SELECT (four + 1.0)/(hundred+1), odd::float8 / (tenthous + 1), format('%s:00:%s:00:%s:00', to_hex(odd), to_hex(even), to_hex(hundred))::macaddr, - substr(md5(unique1::text), 1, 16)::macaddr8, + substr(encode(sha256(unique1::text::bytea), 'hex'), 1, 16)::macaddr8, inet '10.2.3.4' + tenthous, cidr '10.2.3/24' + tenthous, date '1995-08-15' + tenthous, diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out index e06ac93a36..ec6dfe2633 100644 --- a/src/test/regress/expected/compression.out +++ b/src/test/regress/expected/compression.out @@ -102,7 +102,7 @@ SELECT pg_column_compression(f1) FROM cmmove2; -- test externally stored compressed data CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS -'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +$$ select string_agg(encode(sha256(g::text::bytea), 'hex'), '') from generate_series(1, 132) g $$; CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); SELECT pg_column_compression(f1) FROM cmdata2; @@ -123,13 +123,13 @@ SELECT SUBSTR(f1, 200, 5) FROM cmdata1; substr -------- 01234 - 8f14e + 7d4dd (2 rows) SELECT SUBSTR(f1, 200, 5) FROM cmdata2; substr -------- - 8f14e + 7d4dd (1 row) DROP TABLE cmdata2; @@ -315,10 +315,9 @@ SELECT pg_column_compression(f1) FROM cmdata; -- test expression index DROP TABLE cmdata2; -CREATE TABLE cmdata2 (f1 TEXT COMPRESSION pglz, f2 TEXT COMPRESSION lz4); +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz, f2 text COMPRESSION lz4); CREATE UNIQUE INDEX idx1 ON cmdata2 ((f1 || f2)); -INSERT INTO cmdata2 VALUES((SELECT array_agg(md5(g::TEXT))::TEXT FROM -generate_series(1, 50) g), VERSION()); +INSERT INTO cmdata2 VALUES ((SELECT string_agg(encode(sha256(g::text::bytea), 'hex'), '') FROM generate_series(1, 13) g), version()); -- check data is ok SELECT length(f1) FROM cmdata; length @@ -331,7 +330,7 @@ SELECT length(f1) FROM cmdata1; length -------- 10040 - 12449 + 12448 (2 rows) SELECT length(f1) FROM cmmove1; diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out index c0a47646eb..6047c7e0ed 100644 --- a/src/test/regress/expected/compression_1.out +++ b/src/test/regress/expected/compression_1.out @@ -102,7 +102,7 @@ SELECT pg_column_compression(f1) FROM cmmove2; -- test externally stored compressed data CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS -'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +$$ select string_agg(encode(sha256(g::text::bytea), 'hex'), '') from generate_series(1, 132) g $$; CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); SELECT pg_column_compression(f1) FROM cmdata2; @@ -126,7 +126,7 @@ LINE 1: SELECT SUBSTR(f1, 200, 5) FROM cmdata1; SELECT SUBSTR(f1, 200, 5) FROM cmdata2; substr -------- - 8f14e + 7d4dd (1 row) DROP TABLE cmdata2; @@ -307,15 +307,14 @@ SELECT pg_column_compression(f1) FROM cmdata; -- test expression index DROP TABLE cmdata2; -CREATE TABLE cmdata2 (f1 TEXT COMPRESSION pglz, f2 TEXT COMPRESSION lz4); +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz, f2 text COMPRESSION lz4); ERROR: compression method lz4 not supported DETAIL: This functionality requires the server to be built with lz4 support. CREATE UNIQUE INDEX idx1 ON cmdata2 ((f1 || f2)); ERROR: relation "cmdata2" does not exist -INSERT INTO cmdata2 VALUES((SELECT array_agg(md5(g::TEXT))::TEXT FROM -generate_series(1, 50) g), VERSION()); +INSERT INTO cmdata2 VALUES ((SELECT string_agg(encode(sha256(g::text::bytea), 'hex'), '') FROM generate_series(1, 13) g), version()); ERROR: relation "cmdata2" does not exist -LINE 1: INSERT INTO cmdata2 VALUES((SELECT array_agg(md5(g::TEXT))::... +LINE 1: INSERT INTO cmdata2 VALUES ((SELECT string_agg(encode(sha256... ^ -- check data is ok SELECT length(f1) FROM cmdata; diff --git a/src/test/regress/expected/inherit.out b/src/test/regress/expected/inherit.out index 2d49e765de..18509202a9 100644 --- a/src/test/regress/expected/inherit.out +++ b/src/test/regress/expected/inherit.out @@ -2450,7 +2450,7 @@ alter table permtest_child attach partition permtest_grandchild for values in (' alter table permtest_parent attach partition permtest_child for values in (1); create index on permtest_parent (left(c, 3)); insert into permtest_parent - select 1, 'a', left(md5(i::text), 5) from generate_series(0, 100) i; + select 1, 'a', left(encode(sha256(i::text::bytea), 'hex'), 5) from generate_series(0, 100) i; analyze permtest_parent; create role regress_no_child_access; revoke all on permtest_grandchild from regress_no_child_access; diff --git a/src/test/regress/expected/largeobject.out b/src/test/regress/expected/largeobject.out index 31fba2ff9d..5e7b08244c 100644 --- a/src/test/regress/expected/largeobject.out +++ b/src/test/regress/expected/largeobject.out @@ -441,7 +441,7 @@ TRUNCATE lotest_stash_values; \set newloid_1 :LASTOID SELECT lo_from_bytea(0, lo_get(:newloid_1)) AS newloid_2 \gset -SELECT md5(lo_get(:newloid_1)) = md5(lo_get(:newloid_2)); +SELECT sha256(lo_get(:newloid_1)) = sha256(lo_get(:newloid_2)); ?column? ---------- t diff --git a/src/test/regress/expected/largeobject_1.out b/src/test/regress/expected/largeobject_1.out index 7acd7f73e1..46d7388e98 100644 --- a/src/test/regress/expected/largeobject_1.out +++ b/src/test/regress/expected/largeobject_1.out @@ -441,7 +441,7 @@ TRUNCATE lotest_stash_values; \set newloid_1 :LASTOID SELECT lo_from_bytea(0, lo_get(:newloid_1)) AS newloid_2 \gset -SELECT md5(lo_get(:newloid_1)) = md5(lo_get(:newloid_2)); +SELECT sha256(lo_get(:newloid_1)) = sha256(lo_get(:newloid_2)); ?column? ---------- t diff --git a/src/test/regress/expected/matview.out b/src/test/regress/expected/matview.out index 87b6e569a5..fc489a8508 100644 --- a/src/test/regress/expected/matview.out +++ b/src/test/regress/expected/matview.out @@ -556,10 +556,10 @@ SET ROLE regress_user_mvtest; -- duplicate all the aliases used in those queries CREATE TABLE mvtest_foo_data AS SELECT i, i+1 AS tid, - md5(random()::text) AS mv, - md5(random()::text) AS newdata, - md5(random()::text) AS newdata2, - md5(random()::text) AS diff + sha256(random()::text::bytea) AS mv, + sha256(random()::text::bytea) AS newdata, + sha256(random()::text::bytea) AS newdata2, + sha256(random()::text::bytea) AS diff FROM generate_series(1, 10) i; CREATE MATERIALIZED VIEW mvtest_mv_foo AS SELECT * FROM mvtest_foo_data; CREATE MATERIALIZED VIEW mvtest_mv_foo AS SELECT * FROM mvtest_foo_data; diff --git a/src/test/regress/expected/memoize.out b/src/test/regress/expected/memoize.out index 60cbdeec7a..ecdc646985 100644 --- a/src/test/regress/expected/memoize.out +++ b/src/test/regress/expected/memoize.out @@ -164,7 +164,7 @@ DROP TABLE flt; CREATE TABLE strtest (n name, t text); CREATE INDEX strtest_n_idx ON strtest (n); CREATE INDEX strtest_t_idx ON strtest (t); -INSERT INTO strtest VALUES('one','one'),('two','two'),('three',repeat(md5('three'),100)); +INSERT INTO strtest VALUES('one','one'),('two','two'),('three',repeat(encode(sha256('three'),'hex'),50)); -- duplicate rows so we get some cache hits INSERT INTO strtest SELECT * FROM strtest; ANALYZE strtest; diff --git a/src/test/regress/expected/plpgsql.out b/src/test/regress/expected/plpgsql.out index cdc519256a..3dfbfeb7c9 100644 --- a/src/test/regress/expected/plpgsql.out +++ b/src/test/regress/expected/plpgsql.out @@ -3404,22 +3404,22 @@ select * from ret_query1(); create type record_type as (x text, y int, z boolean); create or replace function ret_query2(lim int) returns setof record_type as $$ begin - return query select md5(s.x::text), s.x, s.x > 0 + return query select encode(sha256(s.x::text::bytea), 'hex'), s.x, s.x > 0 from generate_series(-8, lim) s (x) where s.x % 2 = 0; end; $$ language plpgsql; select * from ret_query2(8); - x | y | z -----------------------------------+----+--- - a8d2ec85eaf98407310b72eb73dda247 | -8 | f - 596a3d04481816330f07e4f97510c28f | -6 | f - 0267aaf632e87a63288a08331f22c7c3 | -4 | f - 5d7b9adcbe1c629ec722529dd12e5129 | -2 | f - cfcd208495d565ef66e7dff9f98764da | 0 | f - c81e728d9d4c2f636f067f89cc14862c | 2 | t - a87ff679a2f3e71d9181a67b7542122c | 4 | t - 1679091c5a880faf6fb5e6087eb1b2dc | 6 | t - c9f0f895fb98ab9159f51fd0297e236d | 8 | t + x | y | z +------------------------------------------------------------------+----+--- + e91592205d3881e3ea35d66973bb4898d7282126ab5afcb355c95d2fc17f3822 | -8 | f + 03b26944890929ff751653acb2f2af795cee38f937f379f52ed654a68ce91216 | -6 | f + e5e0093f285a4fb94c3fcc2ad7fd04edd10d429ccda87a9aa5e4718efadf182e | -4 | f + cf3bae39dd692048a8bf961182e6a34dfd323eeb0748e162eaf055107f1cb873 | -2 | f + 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 | 0 | f + d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 | 2 | t + 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a | 4 | t + e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 | 6 | t + 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 | 8 | t (9 rows) -- test EXECUTE USING diff --git a/src/test/regress/expected/rowsecurity.out b/src/test/regress/expected/rowsecurity.out index a415ad168c..7a237ee51b 100644 --- a/src/test/regress/expected/rowsecurity.out +++ b/src/test/regress/expected/rowsecurity.out @@ -1408,9 +1408,9 @@ ERROR: infinite recursion detected in policy for relation "rec1" -- SET SESSION AUTHORIZATION regress_rls_alice; CREATE TABLE s1 (a int, b text); -INSERT INTO s1 (SELECT x, md5(x::text) FROM generate_series(-10,10) x); +INSERT INTO s1 (SELECT x, encode(sha256(x::text::bytea), 'hex') FROM generate_series(-10,10) x); CREATE TABLE s2 (x int, y text); -INSERT INTO s2 (SELECT x, md5(x::text) FROM generate_series(-6,6) x); +INSERT INTO s2 (SELECT x, encode(sha256(x::text::bytea), 'hex') FROM generate_series(-6,6) x); GRANT SELECT ON s1, s2 TO regress_rls_bob; CREATE POLICY p1 ON s1 USING (a in (select x from s2 where y like '%2f%')); CREATE POLICY p2 ON s2 USING (x in (select a from s1 where b like '%22%')); @@ -1428,13 +1428,11 @@ DROP POLICY p3 on s1; ALTER POLICY p2 ON s2 USING (x % 2 = 0); SET SESSION AUTHORIZATION regress_rls_bob; SELECT * FROM s1 WHERE f_leak(b); -- OK -NOTICE: f_leak => c81e728d9d4c2f636f067f89cc14862c -NOTICE: f_leak => a87ff679a2f3e71d9181a67b7542122c - a | b ----+---------------------------------- - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c -(2 rows) +NOTICE: f_leak => 03b26944890929ff751653acb2f2af795cee38f937f379f52ed654a68ce91216 + a | b +----+------------------------------------------------------------------ + -6 | 03b26944890929ff751653acb2f2af795cee38f937f379f52ed654a68ce91216 +(1 row) EXPLAIN (COSTS OFF) SELECT * FROM only s1 WHERE f_leak(b); QUERY PLAN @@ -1450,12 +1448,12 @@ SET SESSION AUTHORIZATION regress_rls_alice; ALTER POLICY p1 ON s1 USING (a in (select x from v2)); -- using VIEW in RLS policy SET SESSION AUTHORIZATION regress_rls_bob; SELECT * FROM s1 WHERE f_leak(b); -- OK -NOTICE: f_leak => 0267aaf632e87a63288a08331f22c7c3 -NOTICE: f_leak => 1679091c5a880faf6fb5e6087eb1b2dc - a | b -----+---------------------------------- - -4 | 0267aaf632e87a63288a08331f22c7c3 - 6 | 1679091c5a880faf6fb5e6087eb1b2dc +NOTICE: f_leak => 03b26944890929ff751653acb2f2af795cee38f937f379f52ed654a68ce91216 +NOTICE: f_leak => cf3bae39dd692048a8bf961182e6a34dfd323eeb0748e162eaf055107f1cb873 + a | b +----+------------------------------------------------------------------ + -6 | 03b26944890929ff751653acb2f2af795cee38f937f379f52ed654a68ce91216 + -2 | cf3bae39dd692048a8bf961182e6a34dfd323eeb0748e162eaf055107f1cb873 (2 rows) EXPLAIN (COSTS OFF) SELECT * FROM s1 WHERE f_leak(b); @@ -1469,12 +1467,11 @@ EXPLAIN (COSTS OFF) SELECT * FROM s1 WHERE f_leak(b); (5 rows) SELECT (SELECT x FROM s1 LIMIT 1) xx, * FROM s2 WHERE y like '%28%'; - xx | x | y -----+----+---------------------------------- - -6 | -6 | 596a3d04481816330f07e4f97510c28f - -4 | -4 | 0267aaf632e87a63288a08331f22c7c3 - 2 | 2 | c81e728d9d4c2f636f067f89cc14862c -(3 rows) + xx | x | y +----+----+------------------------------------------------------------------ + -4 | -4 | e5e0093f285a4fb94c3fcc2ad7fd04edd10d429ccda87a9aa5e4718efadf182e + 4 | 4 | 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a +(2 rows) EXPLAIN (COSTS OFF) SELECT (SELECT x FROM s1 LIMIT 1) xx, * FROM s2 WHERE y like '%28%'; QUERY PLAN @@ -1900,7 +1897,7 @@ NOTICE: f_leak => yyyyyy -- SET SESSION AUTHORIZATION regress_rls_alice; CREATE TABLE b1 (a int, b text); -INSERT INTO b1 (SELECT x, md5(x::text) FROM generate_series(-10,10) x); +INSERT INTO b1 (SELECT x, encode(sha256(x::text::bytea), 'hex') FROM generate_series(-10,10) x); CREATE POLICY p1 ON b1 USING (a % 2 = 0); ALTER TABLE b1 ENABLE ROW LEVEL SECURITY; GRANT ALL ON b1 TO regress_rls_bob; @@ -1918,18 +1915,18 @@ EXPLAIN (COSTS OFF) SELECT * FROM bv1 WHERE f_leak(b); (4 rows) SELECT * FROM bv1 WHERE f_leak(b); -NOTICE: f_leak => c81e728d9d4c2f636f067f89cc14862c -NOTICE: f_leak => a87ff679a2f3e71d9181a67b7542122c -NOTICE: f_leak => 1679091c5a880faf6fb5e6087eb1b2dc -NOTICE: f_leak => c9f0f895fb98ab9159f51fd0297e236d -NOTICE: f_leak => d3d9446802a44259755d38e6d163e820 - a | b -----+---------------------------------- - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 10 | d3d9446802a44259755d38e6d163e820 +NOTICE: f_leak => d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 +NOTICE: f_leak => 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a +NOTICE: f_leak => e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 +NOTICE: f_leak => 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 +NOTICE: f_leak => 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 + a | b +----+------------------------------------------------------------------ + 2 | d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 + 4 | 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a + 6 | e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 + 8 | 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 + 10 | 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 (5 rows) INSERT INTO bv1 VALUES (-1, 'xxx'); -- should fail view WCO @@ -1946,7 +1943,7 @@ EXPLAIN (COSTS OFF) UPDATE bv1 SET b = 'yyy' WHERE a = 4 AND f_leak(b); (3 rows) UPDATE bv1 SET b = 'yyy' WHERE a = 4 AND f_leak(b); -NOTICE: f_leak => a87ff679a2f3e71d9181a67b7542122c +NOTICE: f_leak => 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a EXPLAIN (COSTS OFF) DELETE FROM bv1 WHERE a = 6 AND f_leak(b); QUERY PLAN ----------------------------------------------------------------------- @@ -1956,30 +1953,30 @@ EXPLAIN (COSTS OFF) DELETE FROM bv1 WHERE a = 6 AND f_leak(b); (3 rows) DELETE FROM bv1 WHERE a = 6 AND f_leak(b); -NOTICE: f_leak => 1679091c5a880faf6fb5e6087eb1b2dc +NOTICE: f_leak => e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 SET SESSION AUTHORIZATION regress_rls_alice; SELECT * FROM b1; - a | b ------+---------------------------------- - -10 | 1b0fd9efa5279c4203b7c70233f86dbf - -9 | 252e691406782824eec43d7eadc3d256 - -8 | a8d2ec85eaf98407310b72eb73dda247 - -7 | 74687a12d3915d3c4d83f1af7b3683d5 - -6 | 596a3d04481816330f07e4f97510c28f - -5 | 47c1b025fa18ea96c33fbb6718688c0f - -4 | 0267aaf632e87a63288a08331f22c7c3 - -3 | b3149ecea4628efd23d2f86e5a723472 - -2 | 5d7b9adcbe1c629ec722529dd12e5129 - -1 | 6bb61e3b7bce0931da574d19d1d82c88 - 0 | cfcd208495d565ef66e7dff9f98764da - 1 | c4ca4238a0b923820dcc509a6f75849b - 2 | c81e728d9d4c2f636f067f89cc14862c - 3 | eccbc87e4b5ce2fe28308fd9f2a7baf3 - 5 | e4da3b7fbbce2345d7772b0674a318d5 - 7 | 8f14e45fceea167a5a36dedd4bea2543 - 8 | c9f0f895fb98ab9159f51fd0297e236d - 9 | 45c48cce2e2d7fbdea1afc51c7c6ad26 - 10 | d3d9446802a44259755d38e6d163e820 + a | b +-----+------------------------------------------------------------------ + -10 | c171d4ec282b23db89a99880cd624e9ba2940c1d894783602edab5d7481dc1ea + -9 | d5c534fde62beb89c745a59952c8efed8b7523cbd047e682782e4367de9ea3bf + -8 | e91592205d3881e3ea35d66973bb4898d7282126ab5afcb355c95d2fc17f3822 + -7 | a770d3270c9dcdedf12ed9fd70444f7c8a95c26cae3cae9bd867499090a2f14b + -6 | 03b26944890929ff751653acb2f2af795cee38f937f379f52ed654a68ce91216 + -5 | 37aa1ccf80e481832b2db282d4d4f895ee1e31219b7d0f6aee8dc8968828341b + -4 | e5e0093f285a4fb94c3fcc2ad7fd04edd10d429ccda87a9aa5e4718efadf182e + -3 | 615bdd17c2556f82f384392ea8557f8cc88b03501c759e23093ab0b2a9b5cd48 + -2 | cf3bae39dd692048a8bf961182e6a34dfd323eeb0748e162eaf055107f1cb873 + -1 | 1bad6b8cf97131fceab8543e81f7757195fbb1d36b376ee994ad1cf17699c464 + 0 | 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 + 1 | 6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b + 2 | d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 + 3 | 4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce + 5 | ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d + 7 | 7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451 + 8 | 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 + 9 | 19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7 + 10 | 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 12 | xxx 4 | yyy (21 rows) @@ -3038,41 +3035,41 @@ DROP VIEW rls_sbv; -- Expression structure -- SET SESSION AUTHORIZATION regress_rls_alice; -INSERT INTO y2 (SELECT x, md5(x::text) FROM generate_series(0,20) x); +INSERT INTO y2 (SELECT x, encode(sha256(x::text::bytea), 'hex') FROM generate_series(0,20) x); CREATE POLICY p2 ON y2 USING (a % 3 = 0); CREATE POLICY p3 ON y2 USING (a % 4 = 0); SET SESSION AUTHORIZATION regress_rls_bob; SELECT * FROM y2 WHERE f_leak(b); -NOTICE: f_leak => cfcd208495d565ef66e7dff9f98764da -NOTICE: f_leak => c81e728d9d4c2f636f067f89cc14862c -NOTICE: f_leak => eccbc87e4b5ce2fe28308fd9f2a7baf3 -NOTICE: f_leak => a87ff679a2f3e71d9181a67b7542122c -NOTICE: f_leak => 1679091c5a880faf6fb5e6087eb1b2dc -NOTICE: f_leak => c9f0f895fb98ab9159f51fd0297e236d -NOTICE: f_leak => 45c48cce2e2d7fbdea1afc51c7c6ad26 -NOTICE: f_leak => d3d9446802a44259755d38e6d163e820 -NOTICE: f_leak => c20ad4d76fe97759aa27a0c99bff6710 -NOTICE: f_leak => aab3238922bcc25a6f606eb525ffdc56 -NOTICE: f_leak => 9bf31c7ff062936a96d3c8bd1f8f2ff3 -NOTICE: f_leak => c74d97b01eae257e44aa9d5bade97baf -NOTICE: f_leak => 6f4922f45568161a8cdf4ad2299f6d23 -NOTICE: f_leak => 98f13708210194c475687be6106a3b84 - a | b -----+---------------------------------- - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 3 | eccbc87e4b5ce2fe28308fd9f2a7baf3 - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 9 | 45c48cce2e2d7fbdea1afc51c7c6ad26 - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 15 | 9bf31c7ff062936a96d3c8bd1f8f2ff3 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 +NOTICE: f_leak => 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 +NOTICE: f_leak => d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 +NOTICE: f_leak => 4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce +NOTICE: f_leak => 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a +NOTICE: f_leak => e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 +NOTICE: f_leak => 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 +NOTICE: f_leak => 19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7 +NOTICE: f_leak => 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 +NOTICE: f_leak => 6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918 +NOTICE: f_leak => 8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61 +NOTICE: f_leak => e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb +NOTICE: f_leak => b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9 +NOTICE: f_leak => 4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a +NOTICE: f_leak => f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b + a | b +----+------------------------------------------------------------------ + 0 | 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 + 2 | d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 + 3 | 4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce + 4 | 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a + 6 | e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 + 8 | 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 + 9 | 19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7 + 10 | 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 + 12 | 6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918 + 14 | 8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61 + 15 | e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb + 16 | b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9 + 18 | 4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a + 20 | f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b (14 rows) EXPLAIN (COSTS OFF) SELECT * FROM y2 WHERE f_leak(b); @@ -3107,22 +3104,22 @@ NOTICE: f_leak => abc NOTICE: f_leak => abc NOTICE: f_leak => abc NOTICE: f_leak => abc - a | b -----+---------------------------------- - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 3 | eccbc87e4b5ce2fe28308fd9f2a7baf3 - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 9 | 45c48cce2e2d7fbdea1afc51c7c6ad26 - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 15 | 9bf31c7ff062936a96d3c8bd1f8f2ff3 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 + a | b +----+------------------------------------------------------------------ + 0 | 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 + 2 | d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 + 3 | 4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce + 4 | 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a + 6 | e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 + 8 | 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 + 9 | 19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7 + 10 | 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 + 12 | 6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918 + 14 | 8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61 + 15 | e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb + 16 | b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9 + 18 | 4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a + 20 | f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b (14 rows) EXPLAIN (COSTS OFF) SELECT * FROM y2 WHERE f_leak('abc'); @@ -3156,20 +3153,20 @@ EXPLAIN (COSTS OFF) SELECT * FROM y2 JOIN test_qual_pushdown ON (b = abc) WHERE (7 rows) SELECT * FROM y2 JOIN test_qual_pushdown ON (b = abc) WHERE f_leak(b); -NOTICE: f_leak => cfcd208495d565ef66e7dff9f98764da -NOTICE: f_leak => c81e728d9d4c2f636f067f89cc14862c -NOTICE: f_leak => eccbc87e4b5ce2fe28308fd9f2a7baf3 -NOTICE: f_leak => a87ff679a2f3e71d9181a67b7542122c -NOTICE: f_leak => 1679091c5a880faf6fb5e6087eb1b2dc -NOTICE: f_leak => c9f0f895fb98ab9159f51fd0297e236d -NOTICE: f_leak => 45c48cce2e2d7fbdea1afc51c7c6ad26 -NOTICE: f_leak => d3d9446802a44259755d38e6d163e820 -NOTICE: f_leak => c20ad4d76fe97759aa27a0c99bff6710 -NOTICE: f_leak => aab3238922bcc25a6f606eb525ffdc56 -NOTICE: f_leak => 9bf31c7ff062936a96d3c8bd1f8f2ff3 -NOTICE: f_leak => c74d97b01eae257e44aa9d5bade97baf -NOTICE: f_leak => 6f4922f45568161a8cdf4ad2299f6d23 -NOTICE: f_leak => 98f13708210194c475687be6106a3b84 +NOTICE: f_leak => 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 +NOTICE: f_leak => d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 +NOTICE: f_leak => 4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce +NOTICE: f_leak => 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a +NOTICE: f_leak => e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 +NOTICE: f_leak => 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 +NOTICE: f_leak => 19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7 +NOTICE: f_leak => 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 +NOTICE: f_leak => 6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918 +NOTICE: f_leak => 8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61 +NOTICE: f_leak => e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb +NOTICE: f_leak => b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9 +NOTICE: f_leak => 4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a +NOTICE: f_leak => f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b a | b | abc ---+---+----- (0 rows) @@ -3239,33 +3236,33 @@ CREATE TABLE t1 (a integer, b text); CREATE POLICY p1 ON t1 USING (a % 2 = 0); ALTER TABLE t1 ENABLE ROW LEVEL SECURITY; GRANT ALL ON t1 TO regress_rls_bob; -INSERT INTO t1 (SELECT x, md5(x::text) FROM generate_series(0,20) x); +INSERT INTO t1 (SELECT x, encode(sha256(x::text::bytea), 'hex') FROM generate_series(0,20) x); SET SESSION AUTHORIZATION regress_rls_bob; WITH cte1 AS MATERIALIZED (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1; -NOTICE: f_leak => cfcd208495d565ef66e7dff9f98764da -NOTICE: f_leak => c81e728d9d4c2f636f067f89cc14862c -NOTICE: f_leak => a87ff679a2f3e71d9181a67b7542122c -NOTICE: f_leak => 1679091c5a880faf6fb5e6087eb1b2dc -NOTICE: f_leak => c9f0f895fb98ab9159f51fd0297e236d -NOTICE: f_leak => d3d9446802a44259755d38e6d163e820 -NOTICE: f_leak => c20ad4d76fe97759aa27a0c99bff6710 -NOTICE: f_leak => aab3238922bcc25a6f606eb525ffdc56 -NOTICE: f_leak => c74d97b01eae257e44aa9d5bade97baf -NOTICE: f_leak => 6f4922f45568161a8cdf4ad2299f6d23 -NOTICE: f_leak => 98f13708210194c475687be6106a3b84 - a | b -----+---------------------------------- - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 +NOTICE: f_leak => 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 +NOTICE: f_leak => d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 +NOTICE: f_leak => 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a +NOTICE: f_leak => e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 +NOTICE: f_leak => 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 +NOTICE: f_leak => 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 +NOTICE: f_leak => 6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918 +NOTICE: f_leak => 8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61 +NOTICE: f_leak => b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9 +NOTICE: f_leak => 4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a +NOTICE: f_leak => f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b + a | b +----+------------------------------------------------------------------ + 0 | 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 + 2 | d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 + 4 | 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a + 6 | e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 + 8 | 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 + 10 | 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 + 12 | 6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918 + 14 | 8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61 + 16 | b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9 + 18 | 4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a + 20 | f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b (11 rows) EXPLAIN (COSTS OFF) @@ -3281,19 +3278,19 @@ WITH cte1 AS MATERIALIZED (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1; WITH cte1 AS (UPDATE t1 SET a = a + 1 RETURNING *) SELECT * FROM cte1; --fail ERROR: new row violates row-level security policy for table "t1" WITH cte1 AS (UPDATE t1 SET a = a RETURNING *) SELECT * FROM cte1; --ok - a | b -----+---------------------------------- - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 + a | b +----+------------------------------------------------------------------ + 0 | 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 + 2 | d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 + 4 | 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a + 6 | e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 + 8 | 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 + 10 | 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 + 12 | 6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918 + 14 | 8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61 + 16 | b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9 + 18 | 4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a + 20 | f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b (11 rows) WITH cte1 AS (INSERT INTO t1 VALUES (21, 'Fail') RETURNING *) SELECT * FROM cte1; --fail @@ -3344,19 +3341,19 @@ EXPLAIN (COSTS OFF) INSERT INTO t2 (SELECT * FROM t1); (3 rows) SELECT * FROM t2; - a | b -----+---------------------------------- - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 + a | b +----+------------------------------------------------------------------ + 0 | 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 + 2 | d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 + 4 | 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a + 6 | e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 + 8 | 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 + 10 | 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 + 12 | 6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918 + 14 | 8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61 + 16 | b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9 + 18 | 4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a + 20 | f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b 20 | Success (12 rows) @@ -3368,37 +3365,37 @@ EXPLAIN (COSTS OFF) SELECT * FROM t2; CREATE TABLE t3 AS SELECT * FROM t1; SELECT * FROM t3; - a | b -----+---------------------------------- - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 + a | b +----+------------------------------------------------------------------ + 0 | 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 + 2 | d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 + 4 | 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a + 6 | e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 + 8 | 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 + 10 | 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 + 12 | 6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918 + 14 | 8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61 + 16 | b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9 + 18 | 4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a + 20 | f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b 20 | Success (12 rows) SELECT * INTO t4 FROM t1; SELECT * FROM t4; - a | b -----+---------------------------------- - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 + a | b +----+------------------------------------------------------------------ + 0 | 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 + 2 | d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 + 4 | 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a + 6 | e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 + 8 | 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 + 10 | 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 + 12 | 6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918 + 14 | 8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61 + 16 | b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9 + 18 | 4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a + 20 | f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b 20 | Success (12 rows) @@ -3469,29 +3466,29 @@ ALTER TABLE t1 OWNER TO regress_rls_alice; -- Check that default deny does not apply to superuser. RESET SESSION AUTHORIZATION; SELECT * FROM t1; - a | b -----+---------------------------------- - 1 | c4ca4238a0b923820dcc509a6f75849b - 3 | eccbc87e4b5ce2fe28308fd9f2a7baf3 - 5 | e4da3b7fbbce2345d7772b0674a318d5 - 7 | 8f14e45fceea167a5a36dedd4bea2543 - 9 | 45c48cce2e2d7fbdea1afc51c7c6ad26 - 11 | 6512bd43d9caa6e02c990b0a82652dca - 13 | c51ce410c124a10e0db5e4b97fc2af39 - 15 | 9bf31c7ff062936a96d3c8bd1f8f2ff3 - 17 | 70efdf2ec9b086079795c442636b55fb - 19 | 1f0e3dad99908345f7439f8ffabdffc4 - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 + a | b +----+------------------------------------------------------------------ + 1 | 6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b + 3 | 4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce + 5 | ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d + 7 | 7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451 + 9 | 19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7 + 11 | 4fc82b26aecb47d2868c4efbe3581732a3e7cbcc6c2efb32062c08170a05eeb8 + 13 | 3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278 + 15 | e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb + 17 | 4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3 + 19 | 9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767 + 0 | 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 + 2 | d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 + 4 | 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a + 6 | e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 + 8 | 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 + 10 | 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 + 12 | 6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918 + 14 | 8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61 + 16 | b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9 + 18 | 4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a + 20 | f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b 20 | Success (22 rows) @@ -3504,29 +3501,29 @@ EXPLAIN (COSTS OFF) SELECT * FROM t1; -- Check that default deny does not apply to table owner. SET SESSION AUTHORIZATION regress_rls_alice; SELECT * FROM t1; - a | b -----+---------------------------------- - 1 | c4ca4238a0b923820dcc509a6f75849b - 3 | eccbc87e4b5ce2fe28308fd9f2a7baf3 - 5 | e4da3b7fbbce2345d7772b0674a318d5 - 7 | 8f14e45fceea167a5a36dedd4bea2543 - 9 | 45c48cce2e2d7fbdea1afc51c7c6ad26 - 11 | 6512bd43d9caa6e02c990b0a82652dca - 13 | c51ce410c124a10e0db5e4b97fc2af39 - 15 | 9bf31c7ff062936a96d3c8bd1f8f2ff3 - 17 | 70efdf2ec9b086079795c442636b55fb - 19 | 1f0e3dad99908345f7439f8ffabdffc4 - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 + a | b +----+------------------------------------------------------------------ + 1 | 6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b + 3 | 4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce + 5 | ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d + 7 | 7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451 + 9 | 19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7 + 11 | 4fc82b26aecb47d2868c4efbe3581732a3e7cbcc6c2efb32062c08170a05eeb8 + 13 | 3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278 + 15 | e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb + 17 | 4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3 + 19 | 9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767 + 0 | 5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 + 2 | d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 + 4 | 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a + 6 | e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 + 8 | 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 + 10 | 4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 + 12 | 6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918 + 14 | 8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61 + 16 | b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9 + 18 | 4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a + 20 | f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b 20 | Success (22 rows) @@ -3574,35 +3571,35 @@ CREATE TABLE copy_t (a integer, b text); CREATE POLICY p1 ON copy_t USING (a % 2 = 0); ALTER TABLE copy_t ENABLE ROW LEVEL SECURITY; GRANT ALL ON copy_t TO regress_rls_bob, regress_rls_exempt_user; -INSERT INTO copy_t (SELECT x, md5(x::text) FROM generate_series(0,10) x); +INSERT INTO copy_t (SELECT x, encode(sha256(x::text::bytea), 'hex') FROM generate_series(0,10) x); -- Check COPY TO as Superuser/owner. RESET SESSION AUTHORIZATION; SET row_security TO OFF; COPY (SELECT * FROM copy_t ORDER BY a ASC) TO STDOUT WITH DELIMITER ','; -0,cfcd208495d565ef66e7dff9f98764da -1,c4ca4238a0b923820dcc509a6f75849b -2,c81e728d9d4c2f636f067f89cc14862c -3,eccbc87e4b5ce2fe28308fd9f2a7baf3 -4,a87ff679a2f3e71d9181a67b7542122c -5,e4da3b7fbbce2345d7772b0674a318d5 -6,1679091c5a880faf6fb5e6087eb1b2dc -7,8f14e45fceea167a5a36dedd4bea2543 -8,c9f0f895fb98ab9159f51fd0297e236d -9,45c48cce2e2d7fbdea1afc51c7c6ad26 -10,d3d9446802a44259755d38e6d163e820 +0,5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 +1,6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b +2,d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 +3,4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce +4,4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a +5,ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d +6,e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 +7,7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451 +8,2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 +9,19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7 +10,4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 SET row_security TO ON; COPY (SELECT * FROM copy_t ORDER BY a ASC) TO STDOUT WITH DELIMITER ','; -0,cfcd208495d565ef66e7dff9f98764da -1,c4ca4238a0b923820dcc509a6f75849b -2,c81e728d9d4c2f636f067f89cc14862c -3,eccbc87e4b5ce2fe28308fd9f2a7baf3 -4,a87ff679a2f3e71d9181a67b7542122c -5,e4da3b7fbbce2345d7772b0674a318d5 -6,1679091c5a880faf6fb5e6087eb1b2dc -7,8f14e45fceea167a5a36dedd4bea2543 -8,c9f0f895fb98ab9159f51fd0297e236d -9,45c48cce2e2d7fbdea1afc51c7c6ad26 -10,d3d9446802a44259755d38e6d163e820 +0,5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 +1,6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b +2,d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 +3,4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce +4,4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a +5,ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d +6,e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 +7,7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451 +8,2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 +9,19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7 +10,4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 -- Check COPY TO as user with permissions. SET SESSION AUTHORIZATION regress_rls_bob; SET row_security TO OFF; @@ -3610,40 +3607,40 @@ COPY (SELECT * FROM copy_t ORDER BY a ASC) TO STDOUT WITH DELIMITER ','; --fail ERROR: query would be affected by row-level security policy for table "copy_t" SET row_security TO ON; COPY (SELECT * FROM copy_t ORDER BY a ASC) TO STDOUT WITH DELIMITER ','; --ok -0,cfcd208495d565ef66e7dff9f98764da -2,c81e728d9d4c2f636f067f89cc14862c -4,a87ff679a2f3e71d9181a67b7542122c -6,1679091c5a880faf6fb5e6087eb1b2dc -8,c9f0f895fb98ab9159f51fd0297e236d -10,d3d9446802a44259755d38e6d163e820 +0,5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 +2,d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 +4,4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a +6,e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 +8,2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 +10,4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 -- Check COPY TO as user with permissions and BYPASSRLS SET SESSION AUTHORIZATION regress_rls_exempt_user; SET row_security TO OFF; COPY (SELECT * FROM copy_t ORDER BY a ASC) TO STDOUT WITH DELIMITER ','; --ok -0,cfcd208495d565ef66e7dff9f98764da -1,c4ca4238a0b923820dcc509a6f75849b -2,c81e728d9d4c2f636f067f89cc14862c -3,eccbc87e4b5ce2fe28308fd9f2a7baf3 -4,a87ff679a2f3e71d9181a67b7542122c -5,e4da3b7fbbce2345d7772b0674a318d5 -6,1679091c5a880faf6fb5e6087eb1b2dc -7,8f14e45fceea167a5a36dedd4bea2543 -8,c9f0f895fb98ab9159f51fd0297e236d -9,45c48cce2e2d7fbdea1afc51c7c6ad26 -10,d3d9446802a44259755d38e6d163e820 +0,5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 +1,6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b +2,d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 +3,4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce +4,4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a +5,ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d +6,e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 +7,7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451 +8,2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 +9,19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7 +10,4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 SET row_security TO ON; COPY (SELECT * FROM copy_t ORDER BY a ASC) TO STDOUT WITH DELIMITER ','; --ok -0,cfcd208495d565ef66e7dff9f98764da -1,c4ca4238a0b923820dcc509a6f75849b -2,c81e728d9d4c2f636f067f89cc14862c -3,eccbc87e4b5ce2fe28308fd9f2a7baf3 -4,a87ff679a2f3e71d9181a67b7542122c -5,e4da3b7fbbce2345d7772b0674a318d5 -6,1679091c5a880faf6fb5e6087eb1b2dc -7,8f14e45fceea167a5a36dedd4bea2543 -8,c9f0f895fb98ab9159f51fd0297e236d -9,45c48cce2e2d7fbdea1afc51c7c6ad26 -10,d3d9446802a44259755d38e6d163e820 +0,5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9 +1,6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b +2,d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35 +3,4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce +4,4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a +5,ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d +6,e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683 +7,7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451 +8,2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3 +9,19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7 +10,4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5 -- Check COPY TO as user without permissions. SET row_security TO OFF; SET SESSION AUTHORIZATION regress_rls_carol; SET row_security TO OFF; @@ -3659,15 +3656,15 @@ CREATE TABLE copy_rel_to (a integer, b text); CREATE POLICY p1 ON copy_rel_to USING (a % 2 = 0); ALTER TABLE copy_rel_to ENABLE ROW LEVEL SECURITY; GRANT ALL ON copy_rel_to TO regress_rls_bob, regress_rls_exempt_user; -INSERT INTO copy_rel_to VALUES (1, md5('1')); +INSERT INTO copy_rel_to VALUES (1, encode(sha256('1'), 'hex')); -- Check COPY TO as Superuser/owner. RESET SESSION AUTHORIZATION; SET row_security TO OFF; COPY copy_rel_to TO STDOUT WITH DELIMITER ','; -1,c4ca4238a0b923820dcc509a6f75849b +1,6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b SET row_security TO ON; COPY copy_rel_to TO STDOUT WITH DELIMITER ','; -1,c4ca4238a0b923820dcc509a6f75849b +1,6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b -- Check COPY TO as user with permissions. SET SESSION AUTHORIZATION regress_rls_bob; SET row_security TO OFF; @@ -3679,10 +3676,10 @@ COPY copy_rel_to TO STDOUT WITH DELIMITER ','; --ok SET SESSION AUTHORIZATION regress_rls_exempt_user; SET row_security TO OFF; COPY copy_rel_to TO STDOUT WITH DELIMITER ','; --ok -1,c4ca4238a0b923820dcc509a6f75849b +1,6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b SET row_security TO ON; COPY copy_rel_to TO STDOUT WITH DELIMITER ','; --ok -1,c4ca4238a0b923820dcc509a6f75849b +1,6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b -- Check COPY TO as user without permissions. SET row_security TO OFF; SET SESSION AUTHORIZATION regress_rls_carol; SET row_security TO OFF; diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out index 03880874c1..41d2439388 100644 --- a/src/test/regress/expected/stats_ext.out +++ b/src/test/regress/expected/stats_ext.out @@ -2615,18 +2615,18 @@ CREATE TABLE mcv_lists_uuid ( WITH (autovacuum_enabled = off); INSERT INTO mcv_lists_uuid (a, b, c) SELECT - md5(mod(i,100)::text)::uuid, - md5(mod(i,50)::text)::uuid, - md5(mod(i,25)::text)::uuid + substr(encode(sha256(mod(i,100)::text::bytea), 'hex'), 1, 32)::uuid, + substr(encode(sha256(mod(i,50)::text::bytea), 'hex'), 1, 32)::uuid, + substr(encode(sha256(mod(i,25)::text::bytea), 'hex'), 1, 32)::uuid FROM generate_series(1,5000) s(i); ANALYZE mcv_lists_uuid; -SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND b = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'''); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND b = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'''); estimated | actual -----------+-------- 1 | 50 (1 row) -SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND b = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND c = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'''); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND b = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND c = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'''); estimated | actual -----------+-------- 1 | 50 @@ -2635,13 +2635,13 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''167 CREATE STATISTICS mcv_lists_uuid_stats (mcv) ON a, b, c FROM mcv_lists_uuid; ANALYZE mcv_lists_uuid; -SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND b = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'''); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND b = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'''); estimated | actual -----------+-------- 50 | 50 (1 row) -SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND b = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND c = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'''); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND b = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND c = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'''); estimated | actual -----------+-------- 50 | 50 diff --git a/src/test/regress/sql/arrays.sql b/src/test/regress/sql/arrays.sql index 38e8dd440b..5bdc00d26f 100644 --- a/src/test/regress/sql/arrays.sql +++ b/src/test/regress/sql/arrays.sql @@ -671,20 +671,20 @@ CREATE TEMP TABLE arraggtest ( f1 INT[], f2 TEXT[][], f3 FLOAT[]); -- Check that arrays of composites are safely detoasted when needed -create temp table src (f1 text); +create temp table src (f1 bytea); insert into src - select string_agg(random()::text,'') from generate_series(1,10000); -create type textandtext as (c1 text, c2 text); -create temp table dest (f1 textandtext[]); -insert into dest select array[row(f1,f1)::textandtext] from src; -select length(md5((f1[1]).c2)) from dest; + select string_agg(random()::text::bytea,'') from generate_series(1,10000); +create type byteaandbytea as (c1 bytea, c2 bytea); +create temp table dest (f1 byteaandbytea[]); +insert into dest select array[row(f1,f1)::byteaandbytea] from src; +select length(sha256((f1[1]).c2)) from dest; delete from src; -select length(md5((f1[1]).c2)) from dest; +select length(sha256((f1[1]).c2)) from dest; truncate table src; drop table src; -select length(md5((f1[1]).c2)) from dest; +select length(sha256((f1[1]).c2)) from dest; drop table dest; -drop type textandtext; +drop type byteaandbytea; -- Tests for polymorphic-array form of width_bucket() diff --git a/src/test/regress/sql/brin.sql b/src/test/regress/sql/brin.sql index e68e9e18df..7fa125315f 100644 --- a/src/test/regress/sql/brin.sql +++ b/src/test/regress/sql/brin.sql @@ -476,7 +476,7 @@ CREATE TABLE brintest_3 (a text, b text, c text, d text); -- long random strings (~2000 chars each, so ~6kB for min/max on two -- columns) to trigger toasting -WITH rand_value AS (SELECT string_agg(md5(i::text),'') AS val FROM generate_series(1,60) s(i)) +WITH rand_value AS (SELECT string_agg(encode(sha256(i::text::bytea),'hex'),'') AS val FROM generate_series(1,30) s(i)) INSERT INTO brintest_3 SELECT val, val, val, val FROM rand_value; @@ -495,7 +495,7 @@ CREATE INDEX CONCURRENTLY brin_test_temp_idx ON brintest_3(a); -- retry insert with a different random-looking (but deterministic) value -- the value is different, and so should replace either min or max in the -- brin summary -WITH rand_value AS (SELECT string_agg(md5((-i)::text),'') AS val FROM generate_series(1,60) s(i)) +WITH rand_value AS (SELECT string_agg(encode(sha256((-i)::text::bytea),'hex'),'') AS val FROM generate_series(1,30) s(i)) INSERT INTO brintest_3 SELECT val, val, val, val FROM rand_value; diff --git a/src/test/regress/sql/brin_multi.sql b/src/test/regress/sql/brin_multi.sql index 2189b6ccf4..0e26a8705e 100644 --- a/src/test/regress/sql/brin_multi.sql +++ b/src/test/regress/sql/brin_multi.sql @@ -30,7 +30,7 @@ CREATE TABLE brintest_multi ( (four + 1.0)/(hundred+1), odd::float8 / (tenthous + 1), format('%s:00:%s:00:%s:00', to_hex(odd), to_hex(even), to_hex(hundred))::macaddr, - substr(md5(unique1::text), 1, 16)::macaddr8, + substr(encode(sha256(unique1::text::bytea), 'hex'), 1, 16)::macaddr8, inet '10.2.3.4/24' + tenthous, cidr '10.2.3/24' + tenthous, date '1995-08-15' + tenthous, @@ -182,8 +182,8 @@ CREATE TABLE brinopers_multi (colname name, typ text, '{99, 100, 2, 100, 100}'), ('macaddr8col', 'macaddr8', '{>, >=, =, <=, <}', - '{b1:d1:0e:7b:af:a4:42:12, d9:35:91:bd:f7:86:0e:1e, 72:8f:20:6c:2a:01:bf:57, 23:e8:46:63:86:07:ad:cb, 13:16:8e:6a:2e:6c:84:b4}', - '{33, 15, 1, 13, 6}'), + '{83:f8:14:f7:a9:2e:36:5c, fe:25:92:b4:2a:72:7e:97, 67:93:f9:e2:66:ce:6e:bd, 80:5c:94:a3:58:c1:d4:59, fd:24:28:59:bc:18:ff:bb}', + '{47, 2, 1, 50, 97}'), ('inetcol', 'inet', '{=, <, <=, >, >=}', '{10.2.14.231/24, 255.255.255.255, 255.255.255.255, 0.0.0.0, 0.0.0.0}', @@ -334,7 +334,7 @@ CREATE TABLE brinopers_multi (colname name, typ text, (four + 1.0)/(hundred+1), odd::float8 / (tenthous + 1), format('%s:00:%s:00:%s:00', to_hex(odd), to_hex(even), to_hex(hundred))::macaddr, - substr(md5(unique1::text), 1, 16)::macaddr8, + substr(encode(sha256(unique1::text::bytea), 'hex'), 1, 16)::macaddr8, inet '10.2.3.4' + tenthous, cidr '10.2.3/24' + tenthous, date '1995-08-15' + tenthous, diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql index 86332dcc51..5a45bebad6 100644 --- a/src/test/regress/sql/compression.sql +++ b/src/test/regress/sql/compression.sql @@ -48,7 +48,7 @@ CREATE TABLE cmmove2(f1 text COMPRESSION pglz); -- test externally stored compressed data CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS -'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +$$ select string_agg(encode(sha256(g::text::bytea), 'hex'), '') from generate_series(1, 132) g $$; CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); SELECT pg_column_compression(f1) FROM cmdata2; @@ -133,10 +133,9 @@ CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); -- test expression index DROP TABLE cmdata2; -CREATE TABLE cmdata2 (f1 TEXT COMPRESSION pglz, f2 TEXT COMPRESSION lz4); +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz, f2 text COMPRESSION lz4); CREATE UNIQUE INDEX idx1 ON cmdata2 ((f1 || f2)); -INSERT INTO cmdata2 VALUES((SELECT array_agg(md5(g::TEXT))::TEXT FROM -generate_series(1, 50) g), VERSION()); +INSERT INTO cmdata2 VALUES ((SELECT string_agg(encode(sha256(g::text::bytea), 'hex'), '') FROM generate_series(1, 13) g), version()); -- check data is ok SELECT length(f1) FROM cmdata; diff --git a/src/test/regress/sql/inherit.sql b/src/test/regress/sql/inherit.sql index 195aedb5ff..6c15686d52 100644 --- a/src/test/regress/sql/inherit.sql +++ b/src/test/regress/sql/inherit.sql @@ -882,7 +882,7 @@ CREATE TABLE test_foreign_constraints_inh () INHERITS (test_foreign_constraints) alter table permtest_parent attach partition permtest_child for values in (1); create index on permtest_parent (left(c, 3)); insert into permtest_parent - select 1, 'a', left(md5(i::text), 5) from generate_series(0, 100) i; + select 1, 'a', left(encode(sha256(i::text::bytea), 'hex'), 5) from generate_series(0, 100) i; analyze permtest_parent; create role regress_no_child_access; revoke all on permtest_grandchild from regress_no_child_access; diff --git a/src/test/regress/sql/largeobject.sql b/src/test/regress/sql/largeobject.sql index 15e0dff7a3..207a7d343b 100644 --- a/src/test/regress/sql/largeobject.sql +++ b/src/test/regress/sql/largeobject.sql @@ -244,7 +244,7 @@ CREATE TABLE lotest_stash_values (loid oid, fd integer); SELECT lo_from_bytea(0, lo_get(:newloid_1)) AS newloid_2 \gset -SELECT md5(lo_get(:newloid_1)) = md5(lo_get(:newloid_2)); +SELECT sha256(lo_get(:newloid_1)) = sha256(lo_get(:newloid_2)); SELECT lo_get(:newloid_1, 0, 20); SELECT lo_get(:newloid_1, 10, 20); diff --git a/src/test/regress/sql/matview.sql b/src/test/regress/sql/matview.sql index 68b9ccfd45..09c55ec98d 100644 --- a/src/test/regress/sql/matview.sql +++ b/src/test/regress/sql/matview.sql @@ -216,10 +216,10 @@ CREATE ROLE regress_user_mvtest; -- duplicate all the aliases used in those queries CREATE TABLE mvtest_foo_data AS SELECT i, i+1 AS tid, - md5(random()::text) AS mv, - md5(random()::text) AS newdata, - md5(random()::text) AS newdata2, - md5(random()::text) AS diff + sha256(random()::text::bytea) AS mv, + sha256(random()::text::bytea) AS newdata, + sha256(random()::text::bytea) AS newdata2, + sha256(random()::text::bytea) AS diff FROM generate_series(1, 10) i; CREATE MATERIALIZED VIEW mvtest_mv_foo AS SELECT * FROM mvtest_foo_data; CREATE MATERIALIZED VIEW mvtest_mv_foo AS SELECT * FROM mvtest_foo_data; diff --git a/src/test/regress/sql/memoize.sql b/src/test/regress/sql/memoize.sql index d66acaed85..80870ee8a5 100644 --- a/src/test/regress/sql/memoize.sql +++ b/src/test/regress/sql/memoize.sql @@ -91,7 +91,7 @@ CREATE INDEX flt_f_idx ON flt (f); CREATE TABLE strtest (n name, t text); CREATE INDEX strtest_n_idx ON strtest (n); CREATE INDEX strtest_t_idx ON strtest (t); -INSERT INTO strtest VALUES('one','one'),('two','two'),('three',repeat(md5('three'),100)); +INSERT INTO strtest VALUES('one','one'),('two','two'),('three',repeat(encode(sha256('three'),'hex'),50)); -- duplicate rows so we get some cache hits INSERT INTO strtest SELECT * FROM strtest; ANALYZE strtest; diff --git a/src/test/regress/sql/plpgsql.sql b/src/test/regress/sql/plpgsql.sql index 9a53b15081..87246fb395 100644 --- a/src/test/regress/sql/plpgsql.sql +++ b/src/test/regress/sql/plpgsql.sql @@ -2877,7 +2877,7 @@ CREATE FUNCTION reraise_test() RETURNS void AS $$ create or replace function ret_query2(lim int) returns setof record_type as $$ begin - return query select md5(s.x::text), s.x, s.x > 0 + return query select encode(sha256(s.x::text::bytea), 'hex'), s.x, s.x > 0 from generate_series(-8, lim) s (x) where s.x % 2 = 0; end; $$ language plpgsql; diff --git a/src/test/regress/sql/rowsecurity.sql b/src/test/regress/sql/rowsecurity.sql index b38fa8ed8f..f2ba7b45b1 100644 --- a/src/test/regress/sql/rowsecurity.sql +++ b/src/test/regress/sql/rowsecurity.sql @@ -534,10 +534,10 @@ CREATE POLICY r2 ON rec2 USING (a = (SELECT x FROM rec1v WHERE y = b)); -- SET SESSION AUTHORIZATION regress_rls_alice; CREATE TABLE s1 (a int, b text); -INSERT INTO s1 (SELECT x, md5(x::text) FROM generate_series(-10,10) x); +INSERT INTO s1 (SELECT x, encode(sha256(x::text::bytea), 'hex') FROM generate_series(-10,10) x); CREATE TABLE s2 (x int, y text); -INSERT INTO s2 (SELECT x, md5(x::text) FROM generate_series(-6,6) x); +INSERT INTO s2 (SELECT x, encode(sha256(x::text::bytea), 'hex') FROM generate_series(-6,6) x); GRANT SELECT ON s1, s2 TO regress_rls_bob; @@ -669,7 +669,7 @@ CREATE VIEW v2 AS SELECT * FROM s2 WHERE y like '%af%'; -- SET SESSION AUTHORIZATION regress_rls_alice; CREATE TABLE b1 (a int, b text); -INSERT INTO b1 (SELECT x, md5(x::text) FROM generate_series(-10,10) x); +INSERT INTO b1 (SELECT x, encode(sha256(x::text::bytea), 'hex') FROM generate_series(-10,10) x); CREATE POLICY p1 ON b1 USING (a % 2 = 0); ALTER TABLE b1 ENABLE ROW LEVEL SECURITY; @@ -1269,7 +1269,7 @@ CREATE VIEW rls_sbv WITH (security_barrier) AS -- Expression structure -- SET SESSION AUTHORIZATION regress_rls_alice; -INSERT INTO y2 (SELECT x, md5(x::text) FROM generate_series(0,20) x); +INSERT INTO y2 (SELECT x, encode(sha256(x::text::bytea), 'hex') FROM generate_series(0,20) x); CREATE POLICY p2 ON y2 USING (a % 3 = 0); CREATE POLICY p3 ON y2 USING (a % 4 = 0); @@ -1341,7 +1341,7 @@ CREATE POLICY p1 ON t1 USING (a % 2 = 0); GRANT ALL ON t1 TO regress_rls_bob; -INSERT INTO t1 (SELECT x, md5(x::text) FROM generate_series(0,20) x); +INSERT INTO t1 (SELECT x, encode(sha256(x::text::bytea), 'hex') FROM generate_series(0,20) x); SET SESSION AUTHORIZATION regress_rls_bob; @@ -1473,7 +1473,7 @@ CREATE POLICY p1 ON copy_t USING (a % 2 = 0); GRANT ALL ON copy_t TO regress_rls_bob, regress_rls_exempt_user; -INSERT INTO copy_t (SELECT x, md5(x::text) FROM generate_series(0,10) x); +INSERT INTO copy_t (SELECT x, encode(sha256(x::text::bytea), 'hex') FROM generate_series(0,10) x); -- Check COPY TO as Superuser/owner. RESET SESSION AUTHORIZATION; @@ -1513,7 +1513,7 @@ CREATE POLICY p1 ON copy_rel_to USING (a % 2 = 0); GRANT ALL ON copy_rel_to TO regress_rls_bob, regress_rls_exempt_user; -INSERT INTO copy_rel_to VALUES (1, md5('1')); +INSERT INTO copy_rel_to VALUES (1, encode(sha256('1'), 'hex')); -- Check COPY TO as Superuser/owner. RESET SESSION AUTHORIZATION; diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql index d0d42cd013..19527ed50f 100644 --- a/src/test/regress/sql/stats_ext.sql +++ b/src/test/regress/sql/stats_ext.sql @@ -1283,25 +1283,25 @@ CREATE TABLE mcv_lists_uuid ( INSERT INTO mcv_lists_uuid (a, b, c) SELECT - md5(mod(i,100)::text)::uuid, - md5(mod(i,50)::text)::uuid, - md5(mod(i,25)::text)::uuid + substr(encode(sha256(mod(i,100)::text::bytea), 'hex'), 1, 32)::uuid, + substr(encode(sha256(mod(i,50)::text::bytea), 'hex'), 1, 32)::uuid, + substr(encode(sha256(mod(i,25)::text::bytea), 'hex'), 1, 32)::uuid FROM generate_series(1,5000) s(i); ANALYZE mcv_lists_uuid; -SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND b = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'''); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND b = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'''); -SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND b = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND c = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'''); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND b = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND c = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'''); CREATE STATISTICS mcv_lists_uuid_stats (mcv) ON a, b, c FROM mcv_lists_uuid; ANALYZE mcv_lists_uuid; -SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND b = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'''); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND b = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'''); -SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND b = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND c = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'''); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND b = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND c = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'''); DROP TABLE mcv_lists_uuid; -- 2.39.1 ^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Allow tests to pass in OpenSSL FIPS mode @ 2023-03-04 23:04 Tom Lane <[email protected]> parent: Peter Eisentraut <[email protected]> 0 siblings, 2 replies; 6+ messages in thread From: Tom Lane @ 2023-03-04 23:04 UTC (permalink / raw) To: Peter Eisentraut <[email protected]>; +Cc: Michael Paquier <[email protected]>; pgsql-hackers Peter Eisentraut <[email protected]> writes: > [ v2-0001-Remove-incidental-md5-function-uses-from-main-reg.patch ] I've gone through this and have a modest suggestion: let's invent some wrapper functions around encode(sha256()) to reduce the cosmetic diffs and consequent need for closer study of patch changes. In the attached I called them "notmd5()", but I'm surely not wedded to that name. This also accounts for some relatively recent additions to stats_ext.sql that introduced yet more uses of md5(). This passes for me on a FIPS-enabled Fedora system, with the exception of md5.sql and password.sql. I agree that the right thing for md5.sql is just to add a variant expected-file. password.sql could perhaps use some refactoring so that we don't have two large expected-files to manage. The only other place that perhaps needs discussion is rowsecurity.sql, which has some surprisingly large changes: not only do the random strings change, but there are rowcount differences in some results. I believe this is because there are RLS policy checks and view conditions that actually examine the contents of the "md5" strings, eg CREATE POLICY p1 ON s1 USING (a in (select x from s2 where y like '%2f%')); My recommendation is to just accept those changes as OK and move on. I doubt that anybody checked the existing results line-by-line either. So, once we've done something about md5.sql and password.sql, I think this is committable. regards, tom lane Attachments: [text/x-diff] v3-0001-Remove-incidental-md5-function-uses-from-main-reg.patch (60.0K, ../../[email protected]/2-v3-0001-Remove-incidental-md5-function-uses-from-main-reg.patch) download | inline diff: diff --git a/src/test/regress/expected/arrays.out b/src/test/regress/expected/arrays.out index 0ff54a18de..1c46da1e0b 100644 --- a/src/test/regress/expected/arrays.out +++ b/src/test/regress/expected/arrays.out @@ -2303,14 +2303,14 @@ insert into src create type textandtext as (c1 text, c2 text); create temp table dest (f1 textandtext[]); insert into dest select array[row(f1,f1)::textandtext] from src; -select length(md5((f1[1]).c2)) from dest; +select length(notmd5((f1[1]).c2)) from dest; length -------- 32 (1 row) delete from src; -select length(md5((f1[1]).c2)) from dest; +select length(notmd5((f1[1]).c2)) from dest; length -------- 32 @@ -2318,7 +2318,7 @@ select length(md5((f1[1]).c2)) from dest; truncate table src; drop table src; -select length(md5((f1[1]).c2)) from dest; +select length(notmd5((f1[1]).c2)) from dest; length -------- 32 diff --git a/src/test/regress/expected/brin.out b/src/test/regress/expected/brin.out index 73fa38396e..b3aac15ecc 100644 --- a/src/test/regress/expected/brin.out +++ b/src/test/regress/expected/brin.out @@ -530,7 +530,7 @@ EXPLAIN (COSTS OFF) SELECT * FROM brin_test WHERE b = 1; CREATE TABLE brintest_3 (a text, b text, c text, d text); -- long random strings (~2000 chars each, so ~6kB for min/max on two -- columns) to trigger toasting -WITH rand_value AS (SELECT string_agg(md5(i::text),'') AS val FROM generate_series(1,60) s(i)) +WITH rand_value AS (SELECT string_agg(notmd5(i::text),'') AS val FROM generate_series(1,60) s(i)) INSERT INTO brintest_3 SELECT val, val, val, val FROM rand_value; CREATE INDEX brin_test_toast_idx ON brintest_3 USING brin (b, c); @@ -545,7 +545,7 @@ VACUUM brintest_3; -- retry insert with a different random-looking (but deterministic) value -- the value is different, and so should replace either min or max in the -- brin summary -WITH rand_value AS (SELECT string_agg(md5((-i)::text),'') AS val FROM generate_series(1,60) s(i)) +WITH rand_value AS (SELECT string_agg(notmd5((-i)::text),'') AS val FROM generate_series(1,60) s(i)) INSERT INTO brintest_3 SELECT val, val, val, val FROM rand_value; -- now try some queries, accessing the brin index diff --git a/src/test/regress/expected/brin_multi.out b/src/test/regress/expected/brin_multi.out index f3309f433f..28d136f59c 100644 --- a/src/test/regress/expected/brin_multi.out +++ b/src/test/regress/expected/brin_multi.out @@ -29,7 +29,7 @@ INSERT INTO brintest_multi SELECT (four + 1.0)/(hundred+1), odd::float8 / (tenthous + 1), format('%s:00:%s:00:%s:00', to_hex(odd), to_hex(even), to_hex(hundred))::macaddr, - substr(md5(unique1::text), 1, 16)::macaddr8, + substr(notmd5(unique1::text), 1, 16)::macaddr8, inet '10.2.3.4/24' + tenthous, cidr '10.2.3/24' + tenthous, date '1995-08-15' + tenthous, @@ -179,7 +179,7 @@ INSERT INTO brinopers_multi VALUES ('macaddr8col', 'macaddr8', '{>, >=, =, <=, <}', '{b1:d1:0e:7b:af:a4:42:12, d9:35:91:bd:f7:86:0e:1e, 72:8f:20:6c:2a:01:bf:57, 23:e8:46:63:86:07:ad:cb, 13:16:8e:6a:2e:6c:84:b4}', - '{33, 15, 1, 13, 6}'), + '{31, 17, 1, 11, 4}'), ('inetcol', 'inet', '{=, <, <=, >, >=}', '{10.2.14.231/24, 255.255.255.255, 255.255.255.255, 0.0.0.0, 0.0.0.0}', @@ -327,7 +327,7 @@ INSERT INTO brintest_multi SELECT (four + 1.0)/(hundred+1), odd::float8 / (tenthous + 1), format('%s:00:%s:00:%s:00', to_hex(odd), to_hex(even), to_hex(hundred))::macaddr, - substr(md5(unique1::text), 1, 16)::macaddr8, + substr(notmd5(unique1::text), 1, 16)::macaddr8, inet '10.2.3.4' + tenthous, cidr '10.2.3/24' + tenthous, date '1995-08-15' + tenthous, diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out index e06ac93a36..0e814e3192 100644 --- a/src/test/regress/expected/compression.out +++ b/src/test/regress/expected/compression.out @@ -102,7 +102,7 @@ SELECT pg_column_compression(f1) FROM cmmove2; -- test externally stored compressed data CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS -'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +'select array_agg(notmd5(g::text))::text from generate_series(1, 256) g'; CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); SELECT pg_column_compression(f1) FROM cmdata2; @@ -123,13 +123,13 @@ SELECT SUBSTR(f1, 200, 5) FROM cmdata1; substr -------- 01234 - 8f14e + 79026 (2 rows) SELECT SUBSTR(f1, 200, 5) FROM cmdata2; substr -------- - 8f14e + 79026 (1 row) DROP TABLE cmdata2; @@ -317,7 +317,7 @@ SELECT pg_column_compression(f1) FROM cmdata; DROP TABLE cmdata2; CREATE TABLE cmdata2 (f1 TEXT COMPRESSION pglz, f2 TEXT COMPRESSION lz4); CREATE UNIQUE INDEX idx1 ON cmdata2 ((f1 || f2)); -INSERT INTO cmdata2 VALUES((SELECT array_agg(md5(g::TEXT))::TEXT FROM +INSERT INTO cmdata2 VALUES((SELECT array_agg(notmd5(g::TEXT))::TEXT FROM generate_series(1, 50) g), VERSION()); -- check data is ok SELECT length(f1) FROM cmdata; diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out index c0a47646eb..5fde7bf94f 100644 --- a/src/test/regress/expected/compression_1.out +++ b/src/test/regress/expected/compression_1.out @@ -102,7 +102,7 @@ SELECT pg_column_compression(f1) FROM cmmove2; -- test externally stored compressed data CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS -'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +'select array_agg(notmd5(g::text))::text from generate_series(1, 256) g'; CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); SELECT pg_column_compression(f1) FROM cmdata2; @@ -126,7 +126,7 @@ LINE 1: SELECT SUBSTR(f1, 200, 5) FROM cmdata1; SELECT SUBSTR(f1, 200, 5) FROM cmdata2; substr -------- - 8f14e + 79026 (1 row) DROP TABLE cmdata2; @@ -312,10 +312,10 @@ ERROR: compression method lz4 not supported DETAIL: This functionality requires the server to be built with lz4 support. CREATE UNIQUE INDEX idx1 ON cmdata2 ((f1 || f2)); ERROR: relation "cmdata2" does not exist -INSERT INTO cmdata2 VALUES((SELECT array_agg(md5(g::TEXT))::TEXT FROM +INSERT INTO cmdata2 VALUES((SELECT array_agg(notmd5(g::TEXT))::TEXT FROM generate_series(1, 50) g), VERSION()); ERROR: relation "cmdata2" does not exist -LINE 1: INSERT INTO cmdata2 VALUES((SELECT array_agg(md5(g::TEXT))::... +LINE 1: INSERT INTO cmdata2 VALUES((SELECT array_agg(notmd5(g::TEXT)... ^ -- check data is ok SELECT length(f1) FROM cmdata; diff --git a/src/test/regress/expected/inherit.out b/src/test/regress/expected/inherit.out index e2a0dc80b2..9f98f33b0d 100644 --- a/src/test/regress/expected/inherit.out +++ b/src/test/regress/expected/inherit.out @@ -2560,7 +2560,7 @@ alter table permtest_child attach partition permtest_grandchild for values in (' alter table permtest_parent attach partition permtest_child for values in (1); create index on permtest_parent (left(c, 3)); insert into permtest_parent - select 1, 'a', left(md5(i::text), 5) from generate_series(0, 100) i; + select 1, 'a', left(notmd5(i::text), 5) from generate_series(0, 100) i; analyze permtest_parent; create role regress_no_child_access; revoke all on permtest_grandchild from regress_no_child_access; diff --git a/src/test/regress/expected/largeobject.out b/src/test/regress/expected/largeobject.out index 31fba2ff9d..af65664eb7 100644 --- a/src/test/regress/expected/largeobject.out +++ b/src/test/regress/expected/largeobject.out @@ -441,7 +441,7 @@ TRUNCATE lotest_stash_values; \set newloid_1 :LASTOID SELECT lo_from_bytea(0, lo_get(:newloid_1)) AS newloid_2 \gset -SELECT md5(lo_get(:newloid_1)) = md5(lo_get(:newloid_2)); +SELECT notmd5(lo_get(:newloid_1)) = notmd5(lo_get(:newloid_2)); ?column? ---------- t diff --git a/src/test/regress/expected/largeobject_1.out b/src/test/regress/expected/largeobject_1.out index 7acd7f73e1..10f1782eeb 100644 --- a/src/test/regress/expected/largeobject_1.out +++ b/src/test/regress/expected/largeobject_1.out @@ -441,7 +441,7 @@ TRUNCATE lotest_stash_values; \set newloid_1 :LASTOID SELECT lo_from_bytea(0, lo_get(:newloid_1)) AS newloid_2 \gset -SELECT md5(lo_get(:newloid_1)) = md5(lo_get(:newloid_2)); +SELECT notmd5(lo_get(:newloid_1)) = notmd5(lo_get(:newloid_2)); ?column? ---------- t diff --git a/src/test/regress/expected/matview.out b/src/test/regress/expected/matview.out index 87b6e569a5..8dcf51ccd4 100644 --- a/src/test/regress/expected/matview.out +++ b/src/test/regress/expected/matview.out @@ -556,10 +556,10 @@ SET ROLE regress_user_mvtest; -- duplicate all the aliases used in those queries CREATE TABLE mvtest_foo_data AS SELECT i, i+1 AS tid, - md5(random()::text) AS mv, - md5(random()::text) AS newdata, - md5(random()::text) AS newdata2, - md5(random()::text) AS diff + notmd5(random()::text) AS mv, + notmd5(random()::text) AS newdata, + notmd5(random()::text) AS newdata2, + notmd5(random()::text) AS diff FROM generate_series(1, 10) i; CREATE MATERIALIZED VIEW mvtest_mv_foo AS SELECT * FROM mvtest_foo_data; CREATE MATERIALIZED VIEW mvtest_mv_foo AS SELECT * FROM mvtest_foo_data; diff --git a/src/test/regress/expected/memoize.out b/src/test/regress/expected/memoize.out index 60cbdeec7a..42e4396269 100644 --- a/src/test/regress/expected/memoize.out +++ b/src/test/regress/expected/memoize.out @@ -164,7 +164,7 @@ DROP TABLE flt; CREATE TABLE strtest (n name, t text); CREATE INDEX strtest_n_idx ON strtest (n); CREATE INDEX strtest_t_idx ON strtest (t); -INSERT INTO strtest VALUES('one','one'),('two','two'),('three',repeat(md5('three'),100)); +INSERT INTO strtest VALUES('one','one'),('two','two'),('three',repeat(notmd5('three'),100)); -- duplicate rows so we get some cache hits INSERT INTO strtest SELECT * FROM strtest; ANALYZE strtest; diff --git a/src/test/regress/expected/plpgsql.out b/src/test/regress/expected/plpgsql.out index cdc519256a..8be2fa9b40 100644 --- a/src/test/regress/expected/plpgsql.out +++ b/src/test/regress/expected/plpgsql.out @@ -3404,22 +3404,22 @@ select * from ret_query1(); create type record_type as (x text, y int, z boolean); create or replace function ret_query2(lim int) returns setof record_type as $$ begin - return query select md5(s.x::text), s.x, s.x > 0 + return query select notmd5(s.x::text), s.x, s.x > 0 from generate_series(-8, lim) s (x) where s.x % 2 = 0; end; $$ language plpgsql; select * from ret_query2(8); x | y | z ----------------------------------+----+--- - a8d2ec85eaf98407310b72eb73dda247 | -8 | f - 596a3d04481816330f07e4f97510c28f | -6 | f - 0267aaf632e87a63288a08331f22c7c3 | -4 | f - 5d7b9adcbe1c629ec722529dd12e5129 | -2 | f - cfcd208495d565ef66e7dff9f98764da | 0 | f - c81e728d9d4c2f636f067f89cc14862c | 2 | t - a87ff679a2f3e71d9181a67b7542122c | 4 | t - 1679091c5a880faf6fb5e6087eb1b2dc | 6 | t - c9f0f895fb98ab9159f51fd0297e236d | 8 | t + e91592205d3881e3ea35d66973bb4898 | -8 | f + 03b26944890929ff751653acb2f2af79 | -6 | f + e5e0093f285a4fb94c3fcc2ad7fd04ed | -4 | f + cf3bae39dd692048a8bf961182e6a34d | -2 | f + 5feceb66ffc86f38d952786c6d696c79 | 0 | f + d4735e3a265e16eee03f59718b9b5d03 | 2 | t + 4b227777d4dd1fc61c6f884f48641d02 | 4 | t + e7f6c011776e8db7cd330b54174fd76f | 6 | t + 2c624232cdd221771294dfbb310aca00 | 8 | t (9 rows) -- test EXECUTE USING diff --git a/src/test/regress/expected/rowsecurity.out b/src/test/regress/expected/rowsecurity.out index a415ad168c..2a7160c1b9 100644 --- a/src/test/regress/expected/rowsecurity.out +++ b/src/test/regress/expected/rowsecurity.out @@ -1408,9 +1408,9 @@ ERROR: infinite recursion detected in policy for relation "rec1" -- SET SESSION AUTHORIZATION regress_rls_alice; CREATE TABLE s1 (a int, b text); -INSERT INTO s1 (SELECT x, md5(x::text) FROM generate_series(-10,10) x); +INSERT INTO s1 (SELECT x, public.notmd5(x::text) FROM generate_series(-10,10) x); CREATE TABLE s2 (x int, y text); -INSERT INTO s2 (SELECT x, md5(x::text) FROM generate_series(-6,6) x); +INSERT INTO s2 (SELECT x, public.notmd5(x::text) FROM generate_series(-6,6) x); GRANT SELECT ON s1, s2 TO regress_rls_bob; CREATE POLICY p1 ON s1 USING (a in (select x from s2 where y like '%2f%')); CREATE POLICY p2 ON s2 USING (x in (select a from s1 where b like '%22%')); @@ -1428,13 +1428,11 @@ DROP POLICY p3 on s1; ALTER POLICY p2 ON s2 USING (x % 2 = 0); SET SESSION AUTHORIZATION regress_rls_bob; SELECT * FROM s1 WHERE f_leak(b); -- OK -NOTICE: f_leak => c81e728d9d4c2f636f067f89cc14862c -NOTICE: f_leak => a87ff679a2f3e71d9181a67b7542122c - a | b ----+---------------------------------- - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c -(2 rows) +NOTICE: f_leak => 03b26944890929ff751653acb2f2af79 + a | b +----+---------------------------------- + -6 | 03b26944890929ff751653acb2f2af79 +(1 row) EXPLAIN (COSTS OFF) SELECT * FROM only s1 WHERE f_leak(b); QUERY PLAN @@ -1450,13 +1448,11 @@ SET SESSION AUTHORIZATION regress_rls_alice; ALTER POLICY p1 ON s1 USING (a in (select x from v2)); -- using VIEW in RLS policy SET SESSION AUTHORIZATION regress_rls_bob; SELECT * FROM s1 WHERE f_leak(b); -- OK -NOTICE: f_leak => 0267aaf632e87a63288a08331f22c7c3 -NOTICE: f_leak => 1679091c5a880faf6fb5e6087eb1b2dc +NOTICE: f_leak => 03b26944890929ff751653acb2f2af79 a | b ----+---------------------------------- - -4 | 0267aaf632e87a63288a08331f22c7c3 - 6 | 1679091c5a880faf6fb5e6087eb1b2dc -(2 rows) + -6 | 03b26944890929ff751653acb2f2af79 +(1 row) EXPLAIN (COSTS OFF) SELECT * FROM s1 WHERE f_leak(b); QUERY PLAN @@ -1471,10 +1467,8 @@ EXPLAIN (COSTS OFF) SELECT * FROM s1 WHERE f_leak(b); SELECT (SELECT x FROM s1 LIMIT 1) xx, * FROM s2 WHERE y like '%28%'; xx | x | y ----+----+---------------------------------- - -6 | -6 | 596a3d04481816330f07e4f97510c28f - -4 | -4 | 0267aaf632e87a63288a08331f22c7c3 - 2 | 2 | c81e728d9d4c2f636f067f89cc14862c -(3 rows) + -4 | -4 | e5e0093f285a4fb94c3fcc2ad7fd04ed +(1 row) EXPLAIN (COSTS OFF) SELECT (SELECT x FROM s1 LIMIT 1) xx, * FROM s2 WHERE y like '%28%'; QUERY PLAN @@ -1900,7 +1894,7 @@ NOTICE: f_leak => yyyyyy -- SET SESSION AUTHORIZATION regress_rls_alice; CREATE TABLE b1 (a int, b text); -INSERT INTO b1 (SELECT x, md5(x::text) FROM generate_series(-10,10) x); +INSERT INTO b1 (SELECT x, public.notmd5(x::text) FROM generate_series(-10,10) x); CREATE POLICY p1 ON b1 USING (a % 2 = 0); ALTER TABLE b1 ENABLE ROW LEVEL SECURITY; GRANT ALL ON b1 TO regress_rls_bob; @@ -1918,18 +1912,18 @@ EXPLAIN (COSTS OFF) SELECT * FROM bv1 WHERE f_leak(b); (4 rows) SELECT * FROM bv1 WHERE f_leak(b); -NOTICE: f_leak => c81e728d9d4c2f636f067f89cc14862c -NOTICE: f_leak => a87ff679a2f3e71d9181a67b7542122c -NOTICE: f_leak => 1679091c5a880faf6fb5e6087eb1b2dc -NOTICE: f_leak => c9f0f895fb98ab9159f51fd0297e236d -NOTICE: f_leak => d3d9446802a44259755d38e6d163e820 +NOTICE: f_leak => d4735e3a265e16eee03f59718b9b5d03 +NOTICE: f_leak => 4b227777d4dd1fc61c6f884f48641d02 +NOTICE: f_leak => e7f6c011776e8db7cd330b54174fd76f +NOTICE: f_leak => 2c624232cdd221771294dfbb310aca00 +NOTICE: f_leak => 4a44dc15364204a80fe80e9039455cc1 a | b ----+---------------------------------- - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 10 | d3d9446802a44259755d38e6d163e820 + 2 | d4735e3a265e16eee03f59718b9b5d03 + 4 | 4b227777d4dd1fc61c6f884f48641d02 + 6 | e7f6c011776e8db7cd330b54174fd76f + 8 | 2c624232cdd221771294dfbb310aca00 + 10 | 4a44dc15364204a80fe80e9039455cc1 (5 rows) INSERT INTO bv1 VALUES (-1, 'xxx'); -- should fail view WCO @@ -1946,7 +1940,7 @@ EXPLAIN (COSTS OFF) UPDATE bv1 SET b = 'yyy' WHERE a = 4 AND f_leak(b); (3 rows) UPDATE bv1 SET b = 'yyy' WHERE a = 4 AND f_leak(b); -NOTICE: f_leak => a87ff679a2f3e71d9181a67b7542122c +NOTICE: f_leak => 4b227777d4dd1fc61c6f884f48641d02 EXPLAIN (COSTS OFF) DELETE FROM bv1 WHERE a = 6 AND f_leak(b); QUERY PLAN ----------------------------------------------------------------------- @@ -1956,30 +1950,30 @@ EXPLAIN (COSTS OFF) DELETE FROM bv1 WHERE a = 6 AND f_leak(b); (3 rows) DELETE FROM bv1 WHERE a = 6 AND f_leak(b); -NOTICE: f_leak => 1679091c5a880faf6fb5e6087eb1b2dc +NOTICE: f_leak => e7f6c011776e8db7cd330b54174fd76f SET SESSION AUTHORIZATION regress_rls_alice; SELECT * FROM b1; a | b -----+---------------------------------- - -10 | 1b0fd9efa5279c4203b7c70233f86dbf - -9 | 252e691406782824eec43d7eadc3d256 - -8 | a8d2ec85eaf98407310b72eb73dda247 - -7 | 74687a12d3915d3c4d83f1af7b3683d5 - -6 | 596a3d04481816330f07e4f97510c28f - -5 | 47c1b025fa18ea96c33fbb6718688c0f - -4 | 0267aaf632e87a63288a08331f22c7c3 - -3 | b3149ecea4628efd23d2f86e5a723472 - -2 | 5d7b9adcbe1c629ec722529dd12e5129 - -1 | 6bb61e3b7bce0931da574d19d1d82c88 - 0 | cfcd208495d565ef66e7dff9f98764da - 1 | c4ca4238a0b923820dcc509a6f75849b - 2 | c81e728d9d4c2f636f067f89cc14862c - 3 | eccbc87e4b5ce2fe28308fd9f2a7baf3 - 5 | e4da3b7fbbce2345d7772b0674a318d5 - 7 | 8f14e45fceea167a5a36dedd4bea2543 - 8 | c9f0f895fb98ab9159f51fd0297e236d - 9 | 45c48cce2e2d7fbdea1afc51c7c6ad26 - 10 | d3d9446802a44259755d38e6d163e820 + -10 | c171d4ec282b23db89a99880cd624e9b + -9 | d5c534fde62beb89c745a59952c8efed + -8 | e91592205d3881e3ea35d66973bb4898 + -7 | a770d3270c9dcdedf12ed9fd70444f7c + -6 | 03b26944890929ff751653acb2f2af79 + -5 | 37aa1ccf80e481832b2db282d4d4f895 + -4 | e5e0093f285a4fb94c3fcc2ad7fd04ed + -3 | 615bdd17c2556f82f384392ea8557f8c + -2 | cf3bae39dd692048a8bf961182e6a34d + -1 | 1bad6b8cf97131fceab8543e81f77571 + 0 | 5feceb66ffc86f38d952786c6d696c79 + 1 | 6b86b273ff34fce19d6b804eff5a3f57 + 2 | d4735e3a265e16eee03f59718b9b5d03 + 3 | 4e07408562bedb8b60ce05c1decfe3ad + 5 | ef2d127de37b942baad06145e54b0c61 + 7 | 7902699be42c8a8e46fbbb4501726517 + 8 | 2c624232cdd221771294dfbb310aca00 + 9 | 19581e27de7ced00ff1ce50b2047e7a5 + 10 | 4a44dc15364204a80fe80e9039455cc1 12 | xxx 4 | yyy (21 rows) @@ -3038,41 +3032,41 @@ DROP VIEW rls_sbv; -- Expression structure -- SET SESSION AUTHORIZATION regress_rls_alice; -INSERT INTO y2 (SELECT x, md5(x::text) FROM generate_series(0,20) x); +INSERT INTO y2 (SELECT x, public.notmd5(x::text) FROM generate_series(0,20) x); CREATE POLICY p2 ON y2 USING (a % 3 = 0); CREATE POLICY p3 ON y2 USING (a % 4 = 0); SET SESSION AUTHORIZATION regress_rls_bob; SELECT * FROM y2 WHERE f_leak(b); -NOTICE: f_leak => cfcd208495d565ef66e7dff9f98764da -NOTICE: f_leak => c81e728d9d4c2f636f067f89cc14862c -NOTICE: f_leak => eccbc87e4b5ce2fe28308fd9f2a7baf3 -NOTICE: f_leak => a87ff679a2f3e71d9181a67b7542122c -NOTICE: f_leak => 1679091c5a880faf6fb5e6087eb1b2dc -NOTICE: f_leak => c9f0f895fb98ab9159f51fd0297e236d -NOTICE: f_leak => 45c48cce2e2d7fbdea1afc51c7c6ad26 -NOTICE: f_leak => d3d9446802a44259755d38e6d163e820 -NOTICE: f_leak => c20ad4d76fe97759aa27a0c99bff6710 -NOTICE: f_leak => aab3238922bcc25a6f606eb525ffdc56 -NOTICE: f_leak => 9bf31c7ff062936a96d3c8bd1f8f2ff3 -NOTICE: f_leak => c74d97b01eae257e44aa9d5bade97baf -NOTICE: f_leak => 6f4922f45568161a8cdf4ad2299f6d23 -NOTICE: f_leak => 98f13708210194c475687be6106a3b84 +NOTICE: f_leak => 5feceb66ffc86f38d952786c6d696c79 +NOTICE: f_leak => d4735e3a265e16eee03f59718b9b5d03 +NOTICE: f_leak => 4e07408562bedb8b60ce05c1decfe3ad +NOTICE: f_leak => 4b227777d4dd1fc61c6f884f48641d02 +NOTICE: f_leak => e7f6c011776e8db7cd330b54174fd76f +NOTICE: f_leak => 2c624232cdd221771294dfbb310aca00 +NOTICE: f_leak => 19581e27de7ced00ff1ce50b2047e7a5 +NOTICE: f_leak => 4a44dc15364204a80fe80e9039455cc1 +NOTICE: f_leak => 6b51d431df5d7f141cbececcf79edf3d +NOTICE: f_leak => 8527a891e224136950ff32ca212b45bc +NOTICE: f_leak => e629fa6598d732768f7c726b4b621285 +NOTICE: f_leak => b17ef6d19c7a5b1ee83b907c595526dc +NOTICE: f_leak => 4ec9599fc203d176a301536c2e091a19 +NOTICE: f_leak => f5ca38f748a1d6eaf726b8a42fb575c3 a | b ----+---------------------------------- - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 3 | eccbc87e4b5ce2fe28308fd9f2a7baf3 - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 9 | 45c48cce2e2d7fbdea1afc51c7c6ad26 - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 15 | 9bf31c7ff062936a96d3c8bd1f8f2ff3 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 + 0 | 5feceb66ffc86f38d952786c6d696c79 + 2 | d4735e3a265e16eee03f59718b9b5d03 + 3 | 4e07408562bedb8b60ce05c1decfe3ad + 4 | 4b227777d4dd1fc61c6f884f48641d02 + 6 | e7f6c011776e8db7cd330b54174fd76f + 8 | 2c624232cdd221771294dfbb310aca00 + 9 | 19581e27de7ced00ff1ce50b2047e7a5 + 10 | 4a44dc15364204a80fe80e9039455cc1 + 12 | 6b51d431df5d7f141cbececcf79edf3d + 14 | 8527a891e224136950ff32ca212b45bc + 15 | e629fa6598d732768f7c726b4b621285 + 16 | b17ef6d19c7a5b1ee83b907c595526dc + 18 | 4ec9599fc203d176a301536c2e091a19 + 20 | f5ca38f748a1d6eaf726b8a42fb575c3 (14 rows) EXPLAIN (COSTS OFF) SELECT * FROM y2 WHERE f_leak(b); @@ -3109,20 +3103,20 @@ NOTICE: f_leak => abc NOTICE: f_leak => abc a | b ----+---------------------------------- - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 3 | eccbc87e4b5ce2fe28308fd9f2a7baf3 - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 9 | 45c48cce2e2d7fbdea1afc51c7c6ad26 - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 15 | 9bf31c7ff062936a96d3c8bd1f8f2ff3 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 + 0 | 5feceb66ffc86f38d952786c6d696c79 + 2 | d4735e3a265e16eee03f59718b9b5d03 + 3 | 4e07408562bedb8b60ce05c1decfe3ad + 4 | 4b227777d4dd1fc61c6f884f48641d02 + 6 | e7f6c011776e8db7cd330b54174fd76f + 8 | 2c624232cdd221771294dfbb310aca00 + 9 | 19581e27de7ced00ff1ce50b2047e7a5 + 10 | 4a44dc15364204a80fe80e9039455cc1 + 12 | 6b51d431df5d7f141cbececcf79edf3d + 14 | 8527a891e224136950ff32ca212b45bc + 15 | e629fa6598d732768f7c726b4b621285 + 16 | b17ef6d19c7a5b1ee83b907c595526dc + 18 | 4ec9599fc203d176a301536c2e091a19 + 20 | f5ca38f748a1d6eaf726b8a42fb575c3 (14 rows) EXPLAIN (COSTS OFF) SELECT * FROM y2 WHERE f_leak('abc'); @@ -3156,20 +3150,20 @@ EXPLAIN (COSTS OFF) SELECT * FROM y2 JOIN test_qual_pushdown ON (b = abc) WHERE (7 rows) SELECT * FROM y2 JOIN test_qual_pushdown ON (b = abc) WHERE f_leak(b); -NOTICE: f_leak => cfcd208495d565ef66e7dff9f98764da -NOTICE: f_leak => c81e728d9d4c2f636f067f89cc14862c -NOTICE: f_leak => eccbc87e4b5ce2fe28308fd9f2a7baf3 -NOTICE: f_leak => a87ff679a2f3e71d9181a67b7542122c -NOTICE: f_leak => 1679091c5a880faf6fb5e6087eb1b2dc -NOTICE: f_leak => c9f0f895fb98ab9159f51fd0297e236d -NOTICE: f_leak => 45c48cce2e2d7fbdea1afc51c7c6ad26 -NOTICE: f_leak => d3d9446802a44259755d38e6d163e820 -NOTICE: f_leak => c20ad4d76fe97759aa27a0c99bff6710 -NOTICE: f_leak => aab3238922bcc25a6f606eb525ffdc56 -NOTICE: f_leak => 9bf31c7ff062936a96d3c8bd1f8f2ff3 -NOTICE: f_leak => c74d97b01eae257e44aa9d5bade97baf -NOTICE: f_leak => 6f4922f45568161a8cdf4ad2299f6d23 -NOTICE: f_leak => 98f13708210194c475687be6106a3b84 +NOTICE: f_leak => 5feceb66ffc86f38d952786c6d696c79 +NOTICE: f_leak => d4735e3a265e16eee03f59718b9b5d03 +NOTICE: f_leak => 4e07408562bedb8b60ce05c1decfe3ad +NOTICE: f_leak => 4b227777d4dd1fc61c6f884f48641d02 +NOTICE: f_leak => e7f6c011776e8db7cd330b54174fd76f +NOTICE: f_leak => 2c624232cdd221771294dfbb310aca00 +NOTICE: f_leak => 19581e27de7ced00ff1ce50b2047e7a5 +NOTICE: f_leak => 4a44dc15364204a80fe80e9039455cc1 +NOTICE: f_leak => 6b51d431df5d7f141cbececcf79edf3d +NOTICE: f_leak => 8527a891e224136950ff32ca212b45bc +NOTICE: f_leak => e629fa6598d732768f7c726b4b621285 +NOTICE: f_leak => b17ef6d19c7a5b1ee83b907c595526dc +NOTICE: f_leak => 4ec9599fc203d176a301536c2e091a19 +NOTICE: f_leak => f5ca38f748a1d6eaf726b8a42fb575c3 a | b | abc ---+---+----- (0 rows) @@ -3239,33 +3233,33 @@ CREATE TABLE t1 (a integer, b text); CREATE POLICY p1 ON t1 USING (a % 2 = 0); ALTER TABLE t1 ENABLE ROW LEVEL SECURITY; GRANT ALL ON t1 TO regress_rls_bob; -INSERT INTO t1 (SELECT x, md5(x::text) FROM generate_series(0,20) x); +INSERT INTO t1 (SELECT x, public.notmd5(x::text) FROM generate_series(0,20) x); SET SESSION AUTHORIZATION regress_rls_bob; WITH cte1 AS MATERIALIZED (SELECT * FROM t1 WHERE f_leak(b)) SELECT * FROM cte1; -NOTICE: f_leak => cfcd208495d565ef66e7dff9f98764da -NOTICE: f_leak => c81e728d9d4c2f636f067f89cc14862c -NOTICE: f_leak => a87ff679a2f3e71d9181a67b7542122c -NOTICE: f_leak => 1679091c5a880faf6fb5e6087eb1b2dc -NOTICE: f_leak => c9f0f895fb98ab9159f51fd0297e236d -NOTICE: f_leak => d3d9446802a44259755d38e6d163e820 -NOTICE: f_leak => c20ad4d76fe97759aa27a0c99bff6710 -NOTICE: f_leak => aab3238922bcc25a6f606eb525ffdc56 -NOTICE: f_leak => c74d97b01eae257e44aa9d5bade97baf -NOTICE: f_leak => 6f4922f45568161a8cdf4ad2299f6d23 -NOTICE: f_leak => 98f13708210194c475687be6106a3b84 +NOTICE: f_leak => 5feceb66ffc86f38d952786c6d696c79 +NOTICE: f_leak => d4735e3a265e16eee03f59718b9b5d03 +NOTICE: f_leak => 4b227777d4dd1fc61c6f884f48641d02 +NOTICE: f_leak => e7f6c011776e8db7cd330b54174fd76f +NOTICE: f_leak => 2c624232cdd221771294dfbb310aca00 +NOTICE: f_leak => 4a44dc15364204a80fe80e9039455cc1 +NOTICE: f_leak => 6b51d431df5d7f141cbececcf79edf3d +NOTICE: f_leak => 8527a891e224136950ff32ca212b45bc +NOTICE: f_leak => b17ef6d19c7a5b1ee83b907c595526dc +NOTICE: f_leak => 4ec9599fc203d176a301536c2e091a19 +NOTICE: f_leak => f5ca38f748a1d6eaf726b8a42fb575c3 a | b ----+---------------------------------- - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 + 0 | 5feceb66ffc86f38d952786c6d696c79 + 2 | d4735e3a265e16eee03f59718b9b5d03 + 4 | 4b227777d4dd1fc61c6f884f48641d02 + 6 | e7f6c011776e8db7cd330b54174fd76f + 8 | 2c624232cdd221771294dfbb310aca00 + 10 | 4a44dc15364204a80fe80e9039455cc1 + 12 | 6b51d431df5d7f141cbececcf79edf3d + 14 | 8527a891e224136950ff32ca212b45bc + 16 | b17ef6d19c7a5b1ee83b907c595526dc + 18 | 4ec9599fc203d176a301536c2e091a19 + 20 | f5ca38f748a1d6eaf726b8a42fb575c3 (11 rows) EXPLAIN (COSTS OFF) @@ -3283,17 +3277,17 @@ ERROR: new row violates row-level security policy for table "t1" WITH cte1 AS (UPDATE t1 SET a = a RETURNING *) SELECT * FROM cte1; --ok a | b ----+---------------------------------- - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 + 0 | 5feceb66ffc86f38d952786c6d696c79 + 2 | d4735e3a265e16eee03f59718b9b5d03 + 4 | 4b227777d4dd1fc61c6f884f48641d02 + 6 | e7f6c011776e8db7cd330b54174fd76f + 8 | 2c624232cdd221771294dfbb310aca00 + 10 | 4a44dc15364204a80fe80e9039455cc1 + 12 | 6b51d431df5d7f141cbececcf79edf3d + 14 | 8527a891e224136950ff32ca212b45bc + 16 | b17ef6d19c7a5b1ee83b907c595526dc + 18 | 4ec9599fc203d176a301536c2e091a19 + 20 | f5ca38f748a1d6eaf726b8a42fb575c3 (11 rows) WITH cte1 AS (INSERT INTO t1 VALUES (21, 'Fail') RETURNING *) SELECT * FROM cte1; --fail @@ -3346,17 +3340,17 @@ EXPLAIN (COSTS OFF) INSERT INTO t2 (SELECT * FROM t1); SELECT * FROM t2; a | b ----+---------------------------------- - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 + 0 | 5feceb66ffc86f38d952786c6d696c79 + 2 | d4735e3a265e16eee03f59718b9b5d03 + 4 | 4b227777d4dd1fc61c6f884f48641d02 + 6 | e7f6c011776e8db7cd330b54174fd76f + 8 | 2c624232cdd221771294dfbb310aca00 + 10 | 4a44dc15364204a80fe80e9039455cc1 + 12 | 6b51d431df5d7f141cbececcf79edf3d + 14 | 8527a891e224136950ff32ca212b45bc + 16 | b17ef6d19c7a5b1ee83b907c595526dc + 18 | 4ec9599fc203d176a301536c2e091a19 + 20 | f5ca38f748a1d6eaf726b8a42fb575c3 20 | Success (12 rows) @@ -3370,17 +3364,17 @@ CREATE TABLE t3 AS SELECT * FROM t1; SELECT * FROM t3; a | b ----+---------------------------------- - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 + 0 | 5feceb66ffc86f38d952786c6d696c79 + 2 | d4735e3a265e16eee03f59718b9b5d03 + 4 | 4b227777d4dd1fc61c6f884f48641d02 + 6 | e7f6c011776e8db7cd330b54174fd76f + 8 | 2c624232cdd221771294dfbb310aca00 + 10 | 4a44dc15364204a80fe80e9039455cc1 + 12 | 6b51d431df5d7f141cbececcf79edf3d + 14 | 8527a891e224136950ff32ca212b45bc + 16 | b17ef6d19c7a5b1ee83b907c595526dc + 18 | 4ec9599fc203d176a301536c2e091a19 + 20 | f5ca38f748a1d6eaf726b8a42fb575c3 20 | Success (12 rows) @@ -3388,17 +3382,17 @@ SELECT * INTO t4 FROM t1; SELECT * FROM t4; a | b ----+---------------------------------- - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 + 0 | 5feceb66ffc86f38d952786c6d696c79 + 2 | d4735e3a265e16eee03f59718b9b5d03 + 4 | 4b227777d4dd1fc61c6f884f48641d02 + 6 | e7f6c011776e8db7cd330b54174fd76f + 8 | 2c624232cdd221771294dfbb310aca00 + 10 | 4a44dc15364204a80fe80e9039455cc1 + 12 | 6b51d431df5d7f141cbececcf79edf3d + 14 | 8527a891e224136950ff32ca212b45bc + 16 | b17ef6d19c7a5b1ee83b907c595526dc + 18 | 4ec9599fc203d176a301536c2e091a19 + 20 | f5ca38f748a1d6eaf726b8a42fb575c3 20 | Success (12 rows) @@ -3471,27 +3465,27 @@ RESET SESSION AUTHORIZATION; SELECT * FROM t1; a | b ----+---------------------------------- - 1 | c4ca4238a0b923820dcc509a6f75849b - 3 | eccbc87e4b5ce2fe28308fd9f2a7baf3 - 5 | e4da3b7fbbce2345d7772b0674a318d5 - 7 | 8f14e45fceea167a5a36dedd4bea2543 - 9 | 45c48cce2e2d7fbdea1afc51c7c6ad26 - 11 | 6512bd43d9caa6e02c990b0a82652dca - 13 | c51ce410c124a10e0db5e4b97fc2af39 - 15 | 9bf31c7ff062936a96d3c8bd1f8f2ff3 - 17 | 70efdf2ec9b086079795c442636b55fb - 19 | 1f0e3dad99908345f7439f8ffabdffc4 - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 + 1 | 6b86b273ff34fce19d6b804eff5a3f57 + 3 | 4e07408562bedb8b60ce05c1decfe3ad + 5 | ef2d127de37b942baad06145e54b0c61 + 7 | 7902699be42c8a8e46fbbb4501726517 + 9 | 19581e27de7ced00ff1ce50b2047e7a5 + 11 | 4fc82b26aecb47d2868c4efbe3581732 + 13 | 3fdba35f04dc8c462986c992bcf87554 + 15 | e629fa6598d732768f7c726b4b621285 + 17 | 4523540f1504cd17100c4835e85b7eef + 19 | 9400f1b21cb527d7fa3d3eabba93557a + 0 | 5feceb66ffc86f38d952786c6d696c79 + 2 | d4735e3a265e16eee03f59718b9b5d03 + 4 | 4b227777d4dd1fc61c6f884f48641d02 + 6 | e7f6c011776e8db7cd330b54174fd76f + 8 | 2c624232cdd221771294dfbb310aca00 + 10 | 4a44dc15364204a80fe80e9039455cc1 + 12 | 6b51d431df5d7f141cbececcf79edf3d + 14 | 8527a891e224136950ff32ca212b45bc + 16 | b17ef6d19c7a5b1ee83b907c595526dc + 18 | 4ec9599fc203d176a301536c2e091a19 + 20 | f5ca38f748a1d6eaf726b8a42fb575c3 20 | Success (22 rows) @@ -3506,27 +3500,27 @@ SET SESSION AUTHORIZATION regress_rls_alice; SELECT * FROM t1; a | b ----+---------------------------------- - 1 | c4ca4238a0b923820dcc509a6f75849b - 3 | eccbc87e4b5ce2fe28308fd9f2a7baf3 - 5 | e4da3b7fbbce2345d7772b0674a318d5 - 7 | 8f14e45fceea167a5a36dedd4bea2543 - 9 | 45c48cce2e2d7fbdea1afc51c7c6ad26 - 11 | 6512bd43d9caa6e02c990b0a82652dca - 13 | c51ce410c124a10e0db5e4b97fc2af39 - 15 | 9bf31c7ff062936a96d3c8bd1f8f2ff3 - 17 | 70efdf2ec9b086079795c442636b55fb - 19 | 1f0e3dad99908345f7439f8ffabdffc4 - 0 | cfcd208495d565ef66e7dff9f98764da - 2 | c81e728d9d4c2f636f067f89cc14862c - 4 | a87ff679a2f3e71d9181a67b7542122c - 6 | 1679091c5a880faf6fb5e6087eb1b2dc - 8 | c9f0f895fb98ab9159f51fd0297e236d - 10 | d3d9446802a44259755d38e6d163e820 - 12 | c20ad4d76fe97759aa27a0c99bff6710 - 14 | aab3238922bcc25a6f606eb525ffdc56 - 16 | c74d97b01eae257e44aa9d5bade97baf - 18 | 6f4922f45568161a8cdf4ad2299f6d23 - 20 | 98f13708210194c475687be6106a3b84 + 1 | 6b86b273ff34fce19d6b804eff5a3f57 + 3 | 4e07408562bedb8b60ce05c1decfe3ad + 5 | ef2d127de37b942baad06145e54b0c61 + 7 | 7902699be42c8a8e46fbbb4501726517 + 9 | 19581e27de7ced00ff1ce50b2047e7a5 + 11 | 4fc82b26aecb47d2868c4efbe3581732 + 13 | 3fdba35f04dc8c462986c992bcf87554 + 15 | e629fa6598d732768f7c726b4b621285 + 17 | 4523540f1504cd17100c4835e85b7eef + 19 | 9400f1b21cb527d7fa3d3eabba93557a + 0 | 5feceb66ffc86f38d952786c6d696c79 + 2 | d4735e3a265e16eee03f59718b9b5d03 + 4 | 4b227777d4dd1fc61c6f884f48641d02 + 6 | e7f6c011776e8db7cd330b54174fd76f + 8 | 2c624232cdd221771294dfbb310aca00 + 10 | 4a44dc15364204a80fe80e9039455cc1 + 12 | 6b51d431df5d7f141cbececcf79edf3d + 14 | 8527a891e224136950ff32ca212b45bc + 16 | b17ef6d19c7a5b1ee83b907c595526dc + 18 | 4ec9599fc203d176a301536c2e091a19 + 20 | f5ca38f748a1d6eaf726b8a42fb575c3 20 | Success (22 rows) @@ -3574,35 +3568,35 @@ CREATE TABLE copy_t (a integer, b text); CREATE POLICY p1 ON copy_t USING (a % 2 = 0); ALTER TABLE copy_t ENABLE ROW LEVEL SECURITY; GRANT ALL ON copy_t TO regress_rls_bob, regress_rls_exempt_user; -INSERT INTO copy_t (SELECT x, md5(x::text) FROM generate_series(0,10) x); +INSERT INTO copy_t (SELECT x, public.notmd5(x::text) FROM generate_series(0,10) x); -- Check COPY TO as Superuser/owner. RESET SESSION AUTHORIZATION; SET row_security TO OFF; COPY (SELECT * FROM copy_t ORDER BY a ASC) TO STDOUT WITH DELIMITER ','; -0,cfcd208495d565ef66e7dff9f98764da -1,c4ca4238a0b923820dcc509a6f75849b -2,c81e728d9d4c2f636f067f89cc14862c -3,eccbc87e4b5ce2fe28308fd9f2a7baf3 -4,a87ff679a2f3e71d9181a67b7542122c -5,e4da3b7fbbce2345d7772b0674a318d5 -6,1679091c5a880faf6fb5e6087eb1b2dc -7,8f14e45fceea167a5a36dedd4bea2543 -8,c9f0f895fb98ab9159f51fd0297e236d -9,45c48cce2e2d7fbdea1afc51c7c6ad26 -10,d3d9446802a44259755d38e6d163e820 +0,5feceb66ffc86f38d952786c6d696c79 +1,6b86b273ff34fce19d6b804eff5a3f57 +2,d4735e3a265e16eee03f59718b9b5d03 +3,4e07408562bedb8b60ce05c1decfe3ad +4,4b227777d4dd1fc61c6f884f48641d02 +5,ef2d127de37b942baad06145e54b0c61 +6,e7f6c011776e8db7cd330b54174fd76f +7,7902699be42c8a8e46fbbb4501726517 +8,2c624232cdd221771294dfbb310aca00 +9,19581e27de7ced00ff1ce50b2047e7a5 +10,4a44dc15364204a80fe80e9039455cc1 SET row_security TO ON; COPY (SELECT * FROM copy_t ORDER BY a ASC) TO STDOUT WITH DELIMITER ','; -0,cfcd208495d565ef66e7dff9f98764da -1,c4ca4238a0b923820dcc509a6f75849b -2,c81e728d9d4c2f636f067f89cc14862c -3,eccbc87e4b5ce2fe28308fd9f2a7baf3 -4,a87ff679a2f3e71d9181a67b7542122c -5,e4da3b7fbbce2345d7772b0674a318d5 -6,1679091c5a880faf6fb5e6087eb1b2dc -7,8f14e45fceea167a5a36dedd4bea2543 -8,c9f0f895fb98ab9159f51fd0297e236d -9,45c48cce2e2d7fbdea1afc51c7c6ad26 -10,d3d9446802a44259755d38e6d163e820 +0,5feceb66ffc86f38d952786c6d696c79 +1,6b86b273ff34fce19d6b804eff5a3f57 +2,d4735e3a265e16eee03f59718b9b5d03 +3,4e07408562bedb8b60ce05c1decfe3ad +4,4b227777d4dd1fc61c6f884f48641d02 +5,ef2d127de37b942baad06145e54b0c61 +6,e7f6c011776e8db7cd330b54174fd76f +7,7902699be42c8a8e46fbbb4501726517 +8,2c624232cdd221771294dfbb310aca00 +9,19581e27de7ced00ff1ce50b2047e7a5 +10,4a44dc15364204a80fe80e9039455cc1 -- Check COPY TO as user with permissions. SET SESSION AUTHORIZATION regress_rls_bob; SET row_security TO OFF; @@ -3610,40 +3604,40 @@ COPY (SELECT * FROM copy_t ORDER BY a ASC) TO STDOUT WITH DELIMITER ','; --fail ERROR: query would be affected by row-level security policy for table "copy_t" SET row_security TO ON; COPY (SELECT * FROM copy_t ORDER BY a ASC) TO STDOUT WITH DELIMITER ','; --ok -0,cfcd208495d565ef66e7dff9f98764da -2,c81e728d9d4c2f636f067f89cc14862c -4,a87ff679a2f3e71d9181a67b7542122c -6,1679091c5a880faf6fb5e6087eb1b2dc -8,c9f0f895fb98ab9159f51fd0297e236d -10,d3d9446802a44259755d38e6d163e820 +0,5feceb66ffc86f38d952786c6d696c79 +2,d4735e3a265e16eee03f59718b9b5d03 +4,4b227777d4dd1fc61c6f884f48641d02 +6,e7f6c011776e8db7cd330b54174fd76f +8,2c624232cdd221771294dfbb310aca00 +10,4a44dc15364204a80fe80e9039455cc1 -- Check COPY TO as user with permissions and BYPASSRLS SET SESSION AUTHORIZATION regress_rls_exempt_user; SET row_security TO OFF; COPY (SELECT * FROM copy_t ORDER BY a ASC) TO STDOUT WITH DELIMITER ','; --ok -0,cfcd208495d565ef66e7dff9f98764da -1,c4ca4238a0b923820dcc509a6f75849b -2,c81e728d9d4c2f636f067f89cc14862c -3,eccbc87e4b5ce2fe28308fd9f2a7baf3 -4,a87ff679a2f3e71d9181a67b7542122c -5,e4da3b7fbbce2345d7772b0674a318d5 -6,1679091c5a880faf6fb5e6087eb1b2dc -7,8f14e45fceea167a5a36dedd4bea2543 -8,c9f0f895fb98ab9159f51fd0297e236d -9,45c48cce2e2d7fbdea1afc51c7c6ad26 -10,d3d9446802a44259755d38e6d163e820 +0,5feceb66ffc86f38d952786c6d696c79 +1,6b86b273ff34fce19d6b804eff5a3f57 +2,d4735e3a265e16eee03f59718b9b5d03 +3,4e07408562bedb8b60ce05c1decfe3ad +4,4b227777d4dd1fc61c6f884f48641d02 +5,ef2d127de37b942baad06145e54b0c61 +6,e7f6c011776e8db7cd330b54174fd76f +7,7902699be42c8a8e46fbbb4501726517 +8,2c624232cdd221771294dfbb310aca00 +9,19581e27de7ced00ff1ce50b2047e7a5 +10,4a44dc15364204a80fe80e9039455cc1 SET row_security TO ON; COPY (SELECT * FROM copy_t ORDER BY a ASC) TO STDOUT WITH DELIMITER ','; --ok -0,cfcd208495d565ef66e7dff9f98764da -1,c4ca4238a0b923820dcc509a6f75849b -2,c81e728d9d4c2f636f067f89cc14862c -3,eccbc87e4b5ce2fe28308fd9f2a7baf3 -4,a87ff679a2f3e71d9181a67b7542122c -5,e4da3b7fbbce2345d7772b0674a318d5 -6,1679091c5a880faf6fb5e6087eb1b2dc -7,8f14e45fceea167a5a36dedd4bea2543 -8,c9f0f895fb98ab9159f51fd0297e236d -9,45c48cce2e2d7fbdea1afc51c7c6ad26 -10,d3d9446802a44259755d38e6d163e820 +0,5feceb66ffc86f38d952786c6d696c79 +1,6b86b273ff34fce19d6b804eff5a3f57 +2,d4735e3a265e16eee03f59718b9b5d03 +3,4e07408562bedb8b60ce05c1decfe3ad +4,4b227777d4dd1fc61c6f884f48641d02 +5,ef2d127de37b942baad06145e54b0c61 +6,e7f6c011776e8db7cd330b54174fd76f +7,7902699be42c8a8e46fbbb4501726517 +8,2c624232cdd221771294dfbb310aca00 +9,19581e27de7ced00ff1ce50b2047e7a5 +10,4a44dc15364204a80fe80e9039455cc1 -- Check COPY TO as user without permissions. SET row_security TO OFF; SET SESSION AUTHORIZATION regress_rls_carol; SET row_security TO OFF; @@ -3659,15 +3653,15 @@ CREATE TABLE copy_rel_to (a integer, b text); CREATE POLICY p1 ON copy_rel_to USING (a % 2 = 0); ALTER TABLE copy_rel_to ENABLE ROW LEVEL SECURITY; GRANT ALL ON copy_rel_to TO regress_rls_bob, regress_rls_exempt_user; -INSERT INTO copy_rel_to VALUES (1, md5('1')); +INSERT INTO copy_rel_to VALUES (1, public.notmd5('1')); -- Check COPY TO as Superuser/owner. RESET SESSION AUTHORIZATION; SET row_security TO OFF; COPY copy_rel_to TO STDOUT WITH DELIMITER ','; -1,c4ca4238a0b923820dcc509a6f75849b +1,6b86b273ff34fce19d6b804eff5a3f57 SET row_security TO ON; COPY copy_rel_to TO STDOUT WITH DELIMITER ','; -1,c4ca4238a0b923820dcc509a6f75849b +1,6b86b273ff34fce19d6b804eff5a3f57 -- Check COPY TO as user with permissions. SET SESSION AUTHORIZATION regress_rls_bob; SET row_security TO OFF; @@ -3679,10 +3673,10 @@ COPY copy_rel_to TO STDOUT WITH DELIMITER ','; --ok SET SESSION AUTHORIZATION regress_rls_exempt_user; SET row_security TO OFF; COPY copy_rel_to TO STDOUT WITH DELIMITER ','; --ok -1,c4ca4238a0b923820dcc509a6f75849b +1,6b86b273ff34fce19d6b804eff5a3f57 SET row_security TO ON; COPY copy_rel_to TO STDOUT WITH DELIMITER ','; --ok -1,c4ca4238a0b923820dcc509a6f75849b +1,6b86b273ff34fce19d6b804eff5a3f57 -- Check COPY TO as user without permissions. SET row_security TO OFF; SET SESSION AUTHORIZATION regress_rls_carol; SET row_security TO OFF; diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out index 03880874c1..8777c9fc2b 100644 --- a/src/test/regress/expected/stats_ext.out +++ b/src/test/regress/expected/stats_ext.out @@ -2615,18 +2615,18 @@ CREATE TABLE mcv_lists_uuid ( WITH (autovacuum_enabled = off); INSERT INTO mcv_lists_uuid (a, b, c) SELECT - md5(mod(i,100)::text)::uuid, - md5(mod(i,50)::text)::uuid, - md5(mod(i,25)::text)::uuid + notmd5(mod(i,100)::text)::uuid, + notmd5(mod(i,50)::text)::uuid, + notmd5(mod(i,25)::text)::uuid FROM generate_series(1,5000) s(i); ANALYZE mcv_lists_uuid; -SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND b = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'''); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND b = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'''); estimated | actual -----------+-------- 1 | 50 (1 row) -SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND b = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND c = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'''); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND b = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND c = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'''); estimated | actual -----------+-------- 1 | 50 @@ -2635,13 +2635,13 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''167 CREATE STATISTICS mcv_lists_uuid_stats (mcv) ON a, b, c FROM mcv_lists_uuid; ANALYZE mcv_lists_uuid; -SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND b = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'''); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND b = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'''); estimated | actual -----------+-------- 50 | 50 (1 row) -SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND b = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND c = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'''); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND b = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND c = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'''); estimated | actual -----------+-------- 50 | 50 @@ -2657,7 +2657,7 @@ CREATE TABLE mcv_lists_arrays ( WITH (autovacuum_enabled = off); INSERT INTO mcv_lists_arrays (a, b, c) SELECT - ARRAY[md5((i/100)::text), md5((i/100-1)::text), md5((i/100+1)::text)], + ARRAY[notmd5((i/100)::text), notmd5((i/100-1)::text), notmd5((i/100+1)::text)], ARRAY[(i/100-1)::numeric/1000, (i/100)::numeric/1000, (i/100+1)::numeric/1000], ARRAY[(i/100-1), i/100, (i/100+1)] FROM generate_series(1,5000) s(i); @@ -3038,7 +3038,7 @@ SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND b = DROP TABLE expr_stats; -- statistics on expressions with different data types CREATE TABLE expr_stats (a int, b name, c text); -INSERT INTO expr_stats SELECT mod(i,10), md5(mod(i,10)::text), md5(mod(i,10)::text) FROM generate_series(1,1000) s(i); +INSERT INTO expr_stats SELECT mod(i,10), notmd5(mod(i,10)::text), notmd5(mod(i,10)::text) FROM generate_series(1,1000) s(i); ANALYZE expr_stats; SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND (b || c) <= ''z'' AND (c || b) >= ''0'''); estimated | actual diff --git a/src/test/regress/expected/test_setup.out b/src/test/regress/expected/test_setup.out index 4f54fe20ec..258d4cd60f 100644 --- a/src/test/regress/expected/test_setup.out +++ b/src/test/regress/expected/test_setup.out @@ -231,3 +231,15 @@ create function part_hashtext_length(value text, seed int8) create operator class part_test_text_ops for type text using hash as operator 1 =, function 2 part_hashtext_length(text, int8); +-- +-- These functions are used in tests that used to use md5(), which we now +-- mostly avoid so that the tests will pass in FIPS mode. +-- +create function notmd5(bytea) + returns text + strict immutable parallel safe leakproof + begin atomic; select substr(encode(sha256($1), 'hex'), 1, 32); end; +create function notmd5(text) + returns text + strict immutable parallel safe leakproof + begin atomic; select substr(encode(sha256($1::bytea), 'hex'), 1, 32); end; diff --git a/src/test/regress/sql/arrays.sql b/src/test/regress/sql/arrays.sql index 6ea4dba9f1..39c44ca47a 100644 --- a/src/test/regress/sql/arrays.sql +++ b/src/test/regress/sql/arrays.sql @@ -677,12 +677,12 @@ insert into src create type textandtext as (c1 text, c2 text); create temp table dest (f1 textandtext[]); insert into dest select array[row(f1,f1)::textandtext] from src; -select length(md5((f1[1]).c2)) from dest; +select length(notmd5((f1[1]).c2)) from dest; delete from src; -select length(md5((f1[1]).c2)) from dest; +select length(notmd5((f1[1]).c2)) from dest; truncate table src; drop table src; -select length(md5((f1[1]).c2)) from dest; +select length(notmd5((f1[1]).c2)) from dest; drop table dest; drop type textandtext; diff --git a/src/test/regress/sql/brin.sql b/src/test/regress/sql/brin.sql index e68e9e18df..aea5ed778b 100644 --- a/src/test/regress/sql/brin.sql +++ b/src/test/regress/sql/brin.sql @@ -476,7 +476,7 @@ CREATE TABLE brintest_3 (a text, b text, c text, d text); -- long random strings (~2000 chars each, so ~6kB for min/max on two -- columns) to trigger toasting -WITH rand_value AS (SELECT string_agg(md5(i::text),'') AS val FROM generate_series(1,60) s(i)) +WITH rand_value AS (SELECT string_agg(notmd5(i::text),'') AS val FROM generate_series(1,60) s(i)) INSERT INTO brintest_3 SELECT val, val, val, val FROM rand_value; @@ -495,7 +495,7 @@ VACUUM brintest_3; -- retry insert with a different random-looking (but deterministic) value -- the value is different, and so should replace either min or max in the -- brin summary -WITH rand_value AS (SELECT string_agg(md5((-i)::text),'') AS val FROM generate_series(1,60) s(i)) +WITH rand_value AS (SELECT string_agg(notmd5((-i)::text),'') AS val FROM generate_series(1,60) s(i)) INSERT INTO brintest_3 SELECT val, val, val, val FROM rand_value; diff --git a/src/test/regress/sql/brin_multi.sql b/src/test/regress/sql/brin_multi.sql index 2189b6ccf4..ad67bd8828 100644 --- a/src/test/regress/sql/brin_multi.sql +++ b/src/test/regress/sql/brin_multi.sql @@ -30,7 +30,7 @@ INSERT INTO brintest_multi SELECT (four + 1.0)/(hundred+1), odd::float8 / (tenthous + 1), format('%s:00:%s:00:%s:00', to_hex(odd), to_hex(even), to_hex(hundred))::macaddr, - substr(md5(unique1::text), 1, 16)::macaddr8, + substr(notmd5(unique1::text), 1, 16)::macaddr8, inet '10.2.3.4/24' + tenthous, cidr '10.2.3/24' + tenthous, date '1995-08-15' + tenthous, @@ -183,7 +183,7 @@ INSERT INTO brinopers_multi VALUES ('macaddr8col', 'macaddr8', '{>, >=, =, <=, <}', '{b1:d1:0e:7b:af:a4:42:12, d9:35:91:bd:f7:86:0e:1e, 72:8f:20:6c:2a:01:bf:57, 23:e8:46:63:86:07:ad:cb, 13:16:8e:6a:2e:6c:84:b4}', - '{33, 15, 1, 13, 6}'), + '{31, 17, 1, 11, 4}'), ('inetcol', 'inet', '{=, <, <=, >, >=}', '{10.2.14.231/24, 255.255.255.255, 255.255.255.255, 0.0.0.0, 0.0.0.0}', @@ -334,7 +334,7 @@ INSERT INTO brintest_multi SELECT (four + 1.0)/(hundred+1), odd::float8 / (tenthous + 1), format('%s:00:%s:00:%s:00', to_hex(odd), to_hex(even), to_hex(hundred))::macaddr, - substr(md5(unique1::text), 1, 16)::macaddr8, + substr(notmd5(unique1::text), 1, 16)::macaddr8, inet '10.2.3.4' + tenthous, cidr '10.2.3/24' + tenthous, date '1995-08-15' + tenthous, diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql index 86332dcc51..d8fd3be855 100644 --- a/src/test/regress/sql/compression.sql +++ b/src/test/regress/sql/compression.sql @@ -48,7 +48,7 @@ SELECT pg_column_compression(f1) FROM cmmove2; -- test externally stored compressed data CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS -'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +'select array_agg(notmd5(g::text))::text from generate_series(1, 256) g'; CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); SELECT pg_column_compression(f1) FROM cmdata2; @@ -135,7 +135,7 @@ SELECT pg_column_compression(f1) FROM cmdata; DROP TABLE cmdata2; CREATE TABLE cmdata2 (f1 TEXT COMPRESSION pglz, f2 TEXT COMPRESSION lz4); CREATE UNIQUE INDEX idx1 ON cmdata2 ((f1 || f2)); -INSERT INTO cmdata2 VALUES((SELECT array_agg(md5(g::TEXT))::TEXT FROM +INSERT INTO cmdata2 VALUES((SELECT array_agg(notmd5(g::TEXT))::TEXT FROM generate_series(1, 50) g), VERSION()); -- check data is ok diff --git a/src/test/regress/sql/inherit.sql b/src/test/regress/sql/inherit.sql index 5db6dbc191..211b45b9a6 100644 --- a/src/test/regress/sql/inherit.sql +++ b/src/test/regress/sql/inherit.sql @@ -920,7 +920,7 @@ alter table permtest_child attach partition permtest_grandchild for values in (' alter table permtest_parent attach partition permtest_child for values in (1); create index on permtest_parent (left(c, 3)); insert into permtest_parent - select 1, 'a', left(md5(i::text), 5) from generate_series(0, 100) i; + select 1, 'a', left(notmd5(i::text), 5) from generate_series(0, 100) i; analyze permtest_parent; create role regress_no_child_access; revoke all on permtest_grandchild from regress_no_child_access; diff --git a/src/test/regress/sql/largeobject.sql b/src/test/regress/sql/largeobject.sql index 15e0dff7a3..ae45688ae6 100644 --- a/src/test/regress/sql/largeobject.sql +++ b/src/test/regress/sql/largeobject.sql @@ -244,7 +244,7 @@ TRUNCATE lotest_stash_values; SELECT lo_from_bytea(0, lo_get(:newloid_1)) AS newloid_2 \gset -SELECT md5(lo_get(:newloid_1)) = md5(lo_get(:newloid_2)); +SELECT notmd5(lo_get(:newloid_1)) = notmd5(lo_get(:newloid_2)); SELECT lo_get(:newloid_1, 0, 20); SELECT lo_get(:newloid_1, 10, 20); diff --git a/src/test/regress/sql/matview.sql b/src/test/regress/sql/matview.sql index 68b9ccfd45..4c43899757 100644 --- a/src/test/regress/sql/matview.sql +++ b/src/test/regress/sql/matview.sql @@ -216,10 +216,10 @@ SET ROLE regress_user_mvtest; -- duplicate all the aliases used in those queries CREATE TABLE mvtest_foo_data AS SELECT i, i+1 AS tid, - md5(random()::text) AS mv, - md5(random()::text) AS newdata, - md5(random()::text) AS newdata2, - md5(random()::text) AS diff + notmd5(random()::text) AS mv, + notmd5(random()::text) AS newdata, + notmd5(random()::text) AS newdata2, + notmd5(random()::text) AS diff FROM generate_series(1, 10) i; CREATE MATERIALIZED VIEW mvtest_mv_foo AS SELECT * FROM mvtest_foo_data; CREATE MATERIALIZED VIEW mvtest_mv_foo AS SELECT * FROM mvtest_foo_data; diff --git a/src/test/regress/sql/memoize.sql b/src/test/regress/sql/memoize.sql index d66acaed85..4235c4223c 100644 --- a/src/test/regress/sql/memoize.sql +++ b/src/test/regress/sql/memoize.sql @@ -91,7 +91,7 @@ DROP TABLE flt; CREATE TABLE strtest (n name, t text); CREATE INDEX strtest_n_idx ON strtest (n); CREATE INDEX strtest_t_idx ON strtest (t); -INSERT INTO strtest VALUES('one','one'),('two','two'),('three',repeat(md5('three'),100)); +INSERT INTO strtest VALUES('one','one'),('two','two'),('three',repeat(notmd5('three'),100)); -- duplicate rows so we get some cache hits INSERT INTO strtest SELECT * FROM strtest; ANALYZE strtest; diff --git a/src/test/regress/sql/plpgsql.sql b/src/test/regress/sql/plpgsql.sql index 9a53b15081..58d73761ed 100644 --- a/src/test/regress/sql/plpgsql.sql +++ b/src/test/regress/sql/plpgsql.sql @@ -2877,7 +2877,7 @@ create type record_type as (x text, y int, z boolean); create or replace function ret_query2(lim int) returns setof record_type as $$ begin - return query select md5(s.x::text), s.x, s.x > 0 + return query select notmd5(s.x::text), s.x, s.x > 0 from generate_series(-8, lim) s (x) where s.x % 2 = 0; end; $$ language plpgsql; diff --git a/src/test/regress/sql/rowsecurity.sql b/src/test/regress/sql/rowsecurity.sql index b38fa8ed8f..ce5cf40640 100644 --- a/src/test/regress/sql/rowsecurity.sql +++ b/src/test/regress/sql/rowsecurity.sql @@ -534,10 +534,10 @@ SELECT * FROM rec1; -- fail, mutual recursion via s.b. views -- SET SESSION AUTHORIZATION regress_rls_alice; CREATE TABLE s1 (a int, b text); -INSERT INTO s1 (SELECT x, md5(x::text) FROM generate_series(-10,10) x); +INSERT INTO s1 (SELECT x, public.notmd5(x::text) FROM generate_series(-10,10) x); CREATE TABLE s2 (x int, y text); -INSERT INTO s2 (SELECT x, md5(x::text) FROM generate_series(-6,6) x); +INSERT INTO s2 (SELECT x, public.notmd5(x::text) FROM generate_series(-6,6) x); GRANT SELECT ON s1, s2 TO regress_rls_bob; @@ -669,7 +669,7 @@ DELETE FROM t1 WHERE f_leak(b) RETURNING tableoid::regclass, *, t1; -- SET SESSION AUTHORIZATION regress_rls_alice; CREATE TABLE b1 (a int, b text); -INSERT INTO b1 (SELECT x, md5(x::text) FROM generate_series(-10,10) x); +INSERT INTO b1 (SELECT x, public.notmd5(x::text) FROM generate_series(-10,10) x); CREATE POLICY p1 ON b1 USING (a % 2 = 0); ALTER TABLE b1 ENABLE ROW LEVEL SECURITY; @@ -1269,7 +1269,7 @@ DROP VIEW rls_sbv; -- Expression structure -- SET SESSION AUTHORIZATION regress_rls_alice; -INSERT INTO y2 (SELECT x, md5(x::text) FROM generate_series(0,20) x); +INSERT INTO y2 (SELECT x, public.notmd5(x::text) FROM generate_series(0,20) x); CREATE POLICY p2 ON y2 USING (a % 3 = 0); CREATE POLICY p3 ON y2 USING (a % 4 = 0); @@ -1341,7 +1341,7 @@ ALTER TABLE t1 ENABLE ROW LEVEL SECURITY; GRANT ALL ON t1 TO regress_rls_bob; -INSERT INTO t1 (SELECT x, md5(x::text) FROM generate_series(0,20) x); +INSERT INTO t1 (SELECT x, public.notmd5(x::text) FROM generate_series(0,20) x); SET SESSION AUTHORIZATION regress_rls_bob; @@ -1473,7 +1473,7 @@ ALTER TABLE copy_t ENABLE ROW LEVEL SECURITY; GRANT ALL ON copy_t TO regress_rls_bob, regress_rls_exempt_user; -INSERT INTO copy_t (SELECT x, md5(x::text) FROM generate_series(0,10) x); +INSERT INTO copy_t (SELECT x, public.notmd5(x::text) FROM generate_series(0,10) x); -- Check COPY TO as Superuser/owner. RESET SESSION AUTHORIZATION; @@ -1513,7 +1513,7 @@ ALTER TABLE copy_rel_to ENABLE ROW LEVEL SECURITY; GRANT ALL ON copy_rel_to TO regress_rls_bob, regress_rls_exempt_user; -INSERT INTO copy_rel_to VALUES (1, md5('1')); +INSERT INTO copy_rel_to VALUES (1, public.notmd5('1')); -- Check COPY TO as Superuser/owner. RESET SESSION AUTHORIZATION; diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql index d0d42cd013..b27e9c683a 100644 --- a/src/test/regress/sql/stats_ext.sql +++ b/src/test/regress/sql/stats_ext.sql @@ -1283,25 +1283,25 @@ WITH (autovacuum_enabled = off); INSERT INTO mcv_lists_uuid (a, b, c) SELECT - md5(mod(i,100)::text)::uuid, - md5(mod(i,50)::text)::uuid, - md5(mod(i,25)::text)::uuid + notmd5(mod(i,100)::text)::uuid, + notmd5(mod(i,50)::text)::uuid, + notmd5(mod(i,25)::text)::uuid FROM generate_series(1,5000) s(i); ANALYZE mcv_lists_uuid; -SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND b = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'''); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND b = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'''); -SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND b = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND c = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'''); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND b = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND c = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'''); CREATE STATISTICS mcv_lists_uuid_stats (mcv) ON a, b, c FROM mcv_lists_uuid; ANALYZE mcv_lists_uuid; -SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND b = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'''); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND b = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'''); -SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND b = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'' AND c = ''1679091c-5a88-0faf-6fb5-e6087eb1b2dc'''); +SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_uuid WHERE a = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND b = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'' AND c = ''e7f6c011-776e-8db7-cd33-0b54174fd76f'''); DROP TABLE mcv_lists_uuid; @@ -1315,7 +1315,7 @@ WITH (autovacuum_enabled = off); INSERT INTO mcv_lists_arrays (a, b, c) SELECT - ARRAY[md5((i/100)::text), md5((i/100-1)::text), md5((i/100+1)::text)], + ARRAY[notmd5((i/100)::text), notmd5((i/100-1)::text), notmd5((i/100+1)::text)], ARRAY[(i/100-1)::numeric/1000, (i/100)::numeric/1000, (i/100+1)::numeric/1000], ARRAY[(i/100-1), i/100, (i/100+1)] FROM generate_series(1,5000) s(i); @@ -1515,7 +1515,7 @@ DROP TABLE expr_stats; -- statistics on expressions with different data types CREATE TABLE expr_stats (a int, b name, c text); -INSERT INTO expr_stats SELECT mod(i,10), md5(mod(i,10)::text), md5(mod(i,10)::text) FROM generate_series(1,1000) s(i); +INSERT INTO expr_stats SELECT mod(i,10), notmd5(mod(i,10)::text), notmd5(mod(i,10)::text) FROM generate_series(1,1000) s(i); ANALYZE expr_stats; SELECT * FROM check_estimated_rows('SELECT * FROM expr_stats WHERE a = 0 AND (b || c) <= ''z'' AND (c || b) >= ''0'''); diff --git a/src/test/regress/sql/test_setup.sql b/src/test/regress/sql/test_setup.sql index 8439b38d21..dfb06d4b2e 100644 --- a/src/test/regress/sql/test_setup.sql +++ b/src/test/regress/sql/test_setup.sql @@ -284,3 +284,18 @@ create function part_hashtext_length(value text, seed int8) create operator class part_test_text_ops for type text using hash as operator 1 =, function 2 part_hashtext_length(text, int8); + +-- +-- These functions are used in tests that used to use md5(), which we now +-- mostly avoid so that the tests will pass in FIPS mode. +-- + +create function notmd5(bytea) + returns text + strict immutable parallel safe leakproof + begin atomic; select substr(encode(sha256($1), 'hex'), 1, 32); end; + +create function notmd5(text) + returns text + strict immutable parallel safe leakproof + begin atomic; select substr(encode(sha256($1::bytea), 'hex'), 1, 32); end; ^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Allow tests to pass in OpenSSL FIPS mode @ 2023-03-06 09:02 Daniel Gustafsson <[email protected]> parent: Tom Lane <[email protected]> 1 sibling, 0 replies; 6+ messages in thread From: Daniel Gustafsson @ 2023-03-06 09:02 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; Michael Paquier <[email protected]>; pgsql-hackers > On 5 Mar 2023, at 00:04, Tom Lane <[email protected]> wrote: > > Peter Eisentraut <[email protected]> writes: >> [ v2-0001-Remove-incidental-md5-function-uses-from-main-reg.patch ] > > I've gone through this and have a modest suggestion: let's invent some > wrapper functions around encode(sha256()) to reduce the cosmetic diffs > and consequent need for closer study of patch changes. In the attached > I called them "notmd5()", but I'm surely not wedded to that name. For readers without all context, wouldn't it be better to encode in the function name why we're not just calling a hash like md5? Something like fips_allowed_hash() or similar? -- Daniel Gustafsson ^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Allow tests to pass in OpenSSL FIPS mode @ 2023-03-08 07:34 Peter Eisentraut <[email protected]> parent: Tom Lane <[email protected]> 1 sibling, 1 reply; 6+ messages in thread From: Peter Eisentraut @ 2023-03-08 07:34 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: Michael Paquier <[email protected]>; pgsql-hackers On 05.03.23 00:04, Tom Lane wrote: > I've gone through this and have a modest suggestion: let's invent some > wrapper functions around encode(sha256()) to reduce the cosmetic diffs > and consequent need for closer study of patch changes. In the attached > I called them "notmd5()", but I'm surely not wedded to that name. Do you mean create this on the fly in the test suite, or make it a new built-in function? ^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Allow tests to pass in OpenSSL FIPS mode @ 2023-03-08 07:40 Tom Lane <[email protected]> parent: Peter Eisentraut <[email protected]> 0 siblings, 0 replies; 6+ messages in thread From: Tom Lane @ 2023-03-08 07:40 UTC (permalink / raw) To: Peter Eisentraut <[email protected]>; +Cc: Michael Paquier <[email protected]>; pgsql-hackers Peter Eisentraut <[email protected]> writes: > On 05.03.23 00:04, Tom Lane wrote: >> I've gone through this and have a modest suggestion: let's invent some >> wrapper functions around encode(sha256()) to reduce the cosmetic diffs >> and consequent need for closer study of patch changes. In the attached >> I called them "notmd5()", but I'm surely not wedded to that name. > Do you mean create this on the fly in the test suite, or make it a new > built-in function? The former --- please read my version of the patch. regards, tom lane ^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2023-03-08 07:40 UTC | newest] Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2020-12-03 15:19 [PATCH 3/5] Extended statistics on expressions Tomas Vondra <[email protected]> 2023-01-31 09:55 Re: Allow tests to pass in OpenSSL FIPS mode Peter Eisentraut <[email protected]> 2023-03-04 23:04 ` Re: Allow tests to pass in OpenSSL FIPS mode Tom Lane <[email protected]> 2023-03-06 09:02 ` Re: Allow tests to pass in OpenSSL FIPS mode Daniel Gustafsson <[email protected]> 2023-03-08 07:34 ` Re: Allow tests to pass in OpenSSL FIPS mode Peter Eisentraut <[email protected]> 2023-03-08 07:40 ` Re: Allow tests to pass in OpenSSL FIPS mode Tom Lane <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox