cache lookup errors for missing replication origins

public inbox for [email protected]  
help / color / mirror / Atom feed

cache lookup errors for missing replication origins
30+ messages / 5 participants
[nested] [flat]

* cache lookup errors for missing replication origins
@ 2017-09-05 03:59 Michael Paquier <[email protected]>
  2017-09-06 06:51 ` Re: cache lookup errors for missing replication origins Michael Paquier <[email protected]>
  0 siblings, 1 reply; 30+ messages in thread

From: Michael Paquier @ 2017-09-05 03:59 UTC (permalink / raw)
  To: pgsql-hackers

Hi all,

Cache lookup errors with elog() can be triggered easily by users at
SQL level using replication origin functions:
=# select pg_replication_origin_advance('popo', '0/1');
ERROR:  XX000: cache lookup failed for replication origin 'popo'
LOCATION:  replorigin_by_name, origin.c:229
=# select pg_replication_origin_drop('popo');
ERROR:  XX000: cache lookup failed for replication origin 'popo'
LOCATION:  replorigin_by_name, origin.c:229
=# select pg_replication_origin_session_setup('popo');
ERROR:  XX000: cache lookup failed for replication origin 'popo'
LOCATION:  replorigin_by_name, origin.c:229
=# select pg_replication_origin_progress('popo', true);
ERROR:  XX000: cache lookup failed for replication origin 'popo';
LOCATION:  replorigin_by_name, origin.c:229
A cache lookup means that an illogical status has been reached, but
those code paths don't refer to that. So I think that the error in
replorigin_by_name should be changed to that:
ERROR:  42704: replication slot "%s" does not exist

As far as I can see, replorigin_by_oid makes no use of its missing_ok
= false in the backend code, so letting it untouched would have no
impact. replorigin_by_name with missing_ok = false is only used with
SQL-callable functions, so it could be changed without any impact
elsewhere (without considering externally-maintained replication
modules).

Thanks,
-- 
Michael

-- 
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* Re: cache lookup errors for missing replication origins
  2017-09-05 03:59 cache lookup errors for missing replication origins Michael Paquier <[email protected]>
@ 2017-09-06 06:51 ` Michael Paquier <[email protected]>
  0 siblings, 0 replies; 30+ messages in thread

From: Michael Paquier @ 2017-09-06 06:51 UTC (permalink / raw)
  To: pgsql-hackers

On Tue, Sep 5, 2017 at 12:59 PM, Michael Paquier
<[email protected]> wrote:
> ERROR:  42704: replication slot "%s" does not exist

s/slot/origin/

> As far as I can see, replorigin_by_oid makes no use of its missing_ok
> = false in the backend code, so letting it untouched would have no
> impact. replorigin_by_name with missing_ok = false is only used with
> SQL-callable functions, so it could be changed without any impact
> elsewhere (without considering externally-maintained replication
> modules).

As long as I don't forget, attached is a patch added as well to the
next CF. replorigin_by_oid should have the same switch from elog to
ereport in my opinion. Additional regression tests are included.
-- 
Michael


-- 
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Attachments:

  [application/octet-stream] replorigin-elogs.patch (2.8K, ../../CAB7nPqSsLQA2th_epggz8UKj8rFi2rrd9A5vjo12LF-GP0JOdg@mail.gmail.com/2-replorigin-elogs.patch)
  download | inline diff:
diff --git a/contrib/test_decoding/expected/replorigin.out b/contrib/test_decoding/expected/replorigin.out
index 76d4ea986d..8ea4ddda97 100644
--- a/contrib/test_decoding/expected/replorigin.out
+++ b/contrib/test_decoding/expected/replorigin.out
@@ -26,7 +26,14 @@ SELECT pg_replication_origin_drop('test_decoding: temp');
 (1 row)
 
 SELECT pg_replication_origin_drop('test_decoding: temp');
-ERROR:  cache lookup failed for replication origin 'test_decoding: temp'
+ERROR:  replication origin "test_decoding: temp" does not exist
+-- various failure checks for undefined slots
+select pg_replication_origin_advance('test_decoding: temp', '0/1');
+ERROR:  replication origin "test_decoding: temp" does not exist
+select pg_replication_origin_session_setup('test_decoding: temp');
+ERROR:  replication origin "test_decoding: temp" does not exist
+select pg_replication_origin_progress('test_decoding: temp', true);
+ERROR:  replication origin "test_decoding: temp" does not exist
 SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
  ?column? 
 ----------
diff --git a/contrib/test_decoding/sql/replorigin.sql b/contrib/test_decoding/sql/replorigin.sql
index 7870f0ea32..451cd4bc3b 100644
--- a/contrib/test_decoding/sql/replorigin.sql
+++ b/contrib/test_decoding/sql/replorigin.sql
@@ -13,6 +13,11 @@ SELECT pg_replication_origin_create('test_decoding: temp');
 SELECT pg_replication_origin_drop('test_decoding: temp');
 SELECT pg_replication_origin_drop('test_decoding: temp');
 
+-- various failure checks for undefined slots
+select pg_replication_origin_advance('test_decoding: temp', '0/1');
+select pg_replication_origin_session_setup('test_decoding: temp');
+select pg_replication_origin_progress('test_decoding: temp', true);
+
 SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
 
 -- origin tx
diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c
index edc6efb8a6..1fc8f2dee9 100644
--- a/src/backend/replication/logical/origin.c
+++ b/src/backend/replication/logical/origin.c
@@ -225,8 +225,10 @@ replorigin_by_name(char *roname, bool missing_ok)
 		ReleaseSysCache(tuple);
 	}
 	else if (!missing_ok)
-		elog(ERROR, "cache lookup failed for replication origin '%s'",
-			 roname);
+		ereport(ERROR,
+				(errcode(ERRCODE_UNDEFINED_OBJECT),
+				 errmsg("replication origin \"%s\" does not exist",
+						roname)));
 
 	return roident;
 }
@@ -437,8 +439,10 @@ replorigin_by_oid(RepOriginId roident, bool missing_ok, char **roname)
 		*roname = NULL;
 
 		if (!missing_ok)
-			elog(ERROR, "cache lookup failed for replication origin with oid %u",
-				 roident);
+			ereport(ERROR,
+					(errcode(ERRCODE_UNDEFINED_OBJECT),
+					 errmsg("replication origin with OID %u does not exist",
+							roident)));
 
 		return false;
 	}


^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* [PATCH 1/2] psql \dX: list extended statistics objects
@ 2021-01-07 05:28 Tatsuro Yamada <[email protected]>
  0 siblings, 0 replies; 30+ messages in thread

From: Tatsuro Yamada @ 2021-01-07 05:28 UTC (permalink / raw)

The new command lists extended statistics objects, possibly with their
sizes. All past releases with extended statistics are supported.

Author: Tatsuro Yamada
Reviewed-by: Julien Rouhaud, Alvaro Herrera, Tomas Vondra
Discussion: https://postgr.es/m/c027a541-5856-75a5-0868-341301e1624b%40nttcom.co.jp_1
---
 doc/src/sgml/ref/psql-ref.sgml          |  14 +++
 src/bin/psql/command.c                  |   3 +
 src/bin/psql/describe.c                 | 150 ++++++++++++++++++++++++
 src/bin/psql/describe.h                 |   3 +
 src/bin/psql/help.c                     |   1 +
 src/bin/psql/tab-complete.c             |   4 +-
 src/test/regress/expected/stats_ext.out |  94 +++++++++++++++
 src/test/regress/sql/stats_ext.sql      |  31 +++++
 8 files changed, 299 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml
index 221a967bfe..d01acc92b8 100644
--- a/doc/src/sgml/ref/psql-ref.sgml
+++ b/doc/src/sgml/ref/psql-ref.sgml
@@ -1918,6 +1918,20 @@ testdb=&gt;
         </para>
         </listitem>
       </varlistentry>
+      
+      <varlistentry>
+        <term><literal>\dX [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
+        <listitem>
+        <para>
+        Lists extended statistics.
+        If <replaceable class="parameter">pattern</replaceable>
+        is specified, only those extended statistics whose names match the
+        pattern are listed.
+        If <literal>+</literal> is appended to the command name, each extended
+        statistics is listed with its size.
+        </para>
+        </listitem>
+      </varlistentry>
 
       <varlistentry>
         <term><literal>\dy[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index 303e7c3ad8..c5ebc1c3f4 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -928,6 +928,9 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd)
 				else
 					success = listExtensions(pattern);
 				break;
+			case 'X':			/* Extended Statistics */
+				success = listExtendedStats(pattern, show_verbose);
+				break;
 			case 'y':			/* Event Triggers */
 				success = listEventTriggers(pattern, show_verbose);
 				break;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index caf97563f4..46f54199fb 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -4392,6 +4392,156 @@ listEventTriggers(const char *pattern, bool verbose)
 	return true;
 }
 
+/*
+ * \dX
+ *
+ * Describes extended statistics.
+ */
+bool
+listExtendedStats(const char *pattern, bool verbose)
+{
+	PQExpBufferData buf;
+	PGresult   *res;
+	printQueryOpt myopt = pset.popt;
+
+	if (pset.sversion < 100000)
+	{
+		char		sverbuf[32];
+
+		pg_log_error("The server (version %s) does not support extended statistics.",
+					 formatPGVersionNumber(pset.sversion, false,
+										   sverbuf, sizeof(sverbuf)));
+		return true;
+	}
+
+	initPQExpBuffer(&buf);
+	printfPQExpBuffer(&buf,
+					  "SELECT \n"
+					  "es.stxnamespace::pg_catalog.regnamespace::text AS \"%s\", \n"
+					  "es.stxname AS \"%s\", \n"
+					  "pg_catalog.format('%%s FROM %%s', \n"
+					  "  (SELECT pg_catalog.string_agg(pg_catalog.quote_ident(a.attname),', ') \n"
+					  "   FROM pg_catalog.unnest(es.stxkeys) s(attnum) \n"
+					  "   JOIN pg_catalog.pg_attribute a \n"
+					  "   ON (es.stxrelid = a.attrelid \n"
+					  "   AND a.attnum = s.attnum \n"
+					  "   AND NOT a.attisdropped)), \n"
+					  "es.stxrelid::regclass) AS \"%s\"",
+					  gettext_noop("Schema"),
+					  gettext_noop("Name"),
+					  gettext_noop("Definition"));
+
+	/*
+	 * Since 12 there are two catalogs - one for the definition, one for the
+	 * data built by ANALYZE. Older releases use a single catalog. Also, 12
+	 * adds the MCV statistics kind.
+	 */
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN es.stxdependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"));
+	}
+	else
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxddependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxdmcv IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'm' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"),
+						  gettext_noop("MCV"));
+	}
+
+	/* In verbose mode, print sizes of the extended statistics objects. */
+	if (verbose)
+	{
+		if (pset.sversion < 120000)
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN es.stxdependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxdependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"));
+		}
+		else
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxddependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxddependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxdmcv IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdmcv)::bigint) \n"
+							  "     WHEN 'm' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"),
+							  gettext_noop("MCV_size"));
+		}
+	}
+
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+	else
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "LEFT JOIN pg_catalog.pg_statistic_ext_data esd \n"
+							 "ON es.oid = esd.stxoid \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+
+	processSQLNamePattern(pset.db, &buf, pattern, false,
+						  false, NULL,
+						  "stxname", NULL,
+						  NULL);
+
+	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
+
+	res = PSQLexec(buf.data);
+	termPQExpBuffer(&buf);
+	if (!res)
+		return false;
+
+	myopt.nullPrint = NULL;
+	myopt.title = _("List of extended statistics");
+	myopt.translate_header = true;
+
+	printQuery(res, &myopt, pset.queryFout, false, pset.logfile);
+
+	PQclear(res);
+	return true;
+}
+
 /*
  * \dC
  *
diff --git a/src/bin/psql/describe.h b/src/bin/psql/describe.h
index 6044e3a082..867e57d851 100644
--- a/src/bin/psql/describe.h
+++ b/src/bin/psql/describe.h
@@ -102,6 +102,9 @@ extern bool listExtensions(const char *pattern);
 /* \dx+ */
 extern bool listExtensionContents(const char *pattern);
 
+/* \dX */
+extern bool listExtendedStats(const char *pattern, bool verbose);
+
 /* \dy */
 extern bool listEventTriggers(const char *pattern, bool verbose);
 
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 9ec1c4e810..e42bc8c54e 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -267,6 +267,7 @@ slashUsage(unsigned short int pager)
 	fprintf(output, _("  \\du[S+] [PATTERN]      list roles\n"));
 	fprintf(output, _("  \\dv[S+] [PATTERN]      list views\n"));
 	fprintf(output, _("  \\dx[+]  [PATTERN]      list extensions\n"));
+	fprintf(output, _("  \\dX[+]  [PATTERN]      list extended statistics\n"));
 	fprintf(output, _("  \\dy     [PATTERN]      list event triggers\n"));
 	fprintf(output, _("  \\l[+]   [PATTERN]      list databases\n"));
 	fprintf(output, _("  \\sf[+]  FUNCNAME       show a function's definition\n"));
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9dcab0d2fa..611f1efb15 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1500,7 +1500,7 @@ psql_completion(const char *text, int start, int end)
 		"\\dF", "\\dFd", "\\dFp", "\\dFt", "\\dg", "\\di", "\\dl", "\\dL",
 		"\\dm", "\\dn", "\\do", "\\dO", "\\dp", "\\dP", "\\dPi", "\\dPt",
 		"\\drds", "\\dRs", "\\dRp", "\\ds", "\\dS",
-		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dy",
+		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dX", "\\dy",
 		"\\e", "\\echo", "\\ef", "\\elif", "\\else", "\\encoding",
 		"\\endif", "\\errverbose", "\\ev",
 		"\\f",
@@ -3910,6 +3910,8 @@ psql_completion(const char *text, int start, int end)
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_views, NULL);
 	else if (TailMatchesCS("\\dx*"))
 		COMPLETE_WITH_QUERY(Query_for_list_of_extensions);
+	else if (TailMatchesCS("\\dX*"))
+		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_statistics, NULL);
 	else if (TailMatchesCS("\\dm*"))
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_matviews, NULL);
 	else if (TailMatchesCS("\\dE*"))
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 7bfeaf85f0..8c8a0afcf6 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -1725,6 +1725,100 @@ INSERT INTO tststats.priv_test_tbl
 CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
   FROM tststats.priv_test_tbl;
 ANALYZE tststats.priv_test_tbl;
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+\dX
+                                          List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   
+----------+------------------------+--------------------------------------+-----------+--------------+---------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built
+(12 rows)
+
+\dX stts_?
+                       List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   
+--------+--------+-------------------+-----------+--------------+---------
+ public | stts_1 | a, b FROM stts_t1 | built     |              | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined
+(4 rows)
+
+\dX *stts_hoge
+                               List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   
+--------+-----------+-------------------------------+-----------+--------------+---------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined
+(1 row)
+
+\dX+
+                                                                   List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+----------+------------------------+--------------------------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        |         |                | 106 bytes         | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built   |                |                   | 24 kB
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built   |                |                   | 386 bytes
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built   |                |                   | 294 bytes
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              |         | 13 bytes       |                   | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined |                | 0 bytes           | 0 bytes
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built   |                |                   | 686 bytes
+(12 rows)
+
+\dX+ stts_?
+                                                List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+--------+--------+-------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public | stts_1 | a, b FROM stts_t1 | built     |              |         | 13 bytes       |                   | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(4 rows)
+
+\dX+ *stts_hoge
+                                                       List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+--------+-----------+-------------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(1 row)
+
+\dX+ stts_s2.stts_yama
+                                                    List of extended statistics
+ Schema  |   Name    |       Definition        | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+---------+-----------+-------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ stts_s2 | stts_yama | col1, col3 FROM stts_t3 |           | defined      | defined |                | 0 bytes           | 0 bytes
+(1 row)
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 7912e733ae..db6e3e1ba3 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -912,6 +912,37 @@ CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
 
 ANALYZE tststats.priv_test_tbl;
 
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+
+\dX
+\dX stts_?
+\dX *stts_hoge
+\dX+
+\dX+ stts_?
+\dX+ *stts_hoge
+\dX+ stts_s2.stts_yama
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
+
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
-- 
2.26.2


--------------B0278A9E952F6668D2DBEBA1
Content-Type: text/x-patch; charset=UTF-8;
 name="0002-fixup-rename-defined-to-requested-20210109.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="0002-fixup-rename-defined-to-requested-20210109.patch"



^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* [PATCH 1/2] psql \dX: list extended statistics objects
@ 2021-01-07 05:28 Tatsuro Yamada <[email protected]>
  0 siblings, 0 replies; 30+ messages in thread

From: Tatsuro Yamada @ 2021-01-07 05:28 UTC (permalink / raw)

The new command lists extended statistics objects, possibly with their
sizes. All past releases with extended statistics are supported.

Author: Tatsuro Yamada
Reviewed-by: Julien Rouhaud, Alvaro Herrera, Tomas Vondra
Discussion: https://postgr.es/m/c027a541-5856-75a5-0868-341301e1624b%40nttcom.co.jp_1
---
 doc/src/sgml/ref/psql-ref.sgml          |  14 +++
 src/bin/psql/command.c                  |   3 +
 src/bin/psql/describe.c                 | 150 ++++++++++++++++++++++++
 src/bin/psql/describe.h                 |   3 +
 src/bin/psql/help.c                     |   1 +
 src/bin/psql/tab-complete.c             |   4 +-
 src/test/regress/expected/stats_ext.out |  94 +++++++++++++++
 src/test/regress/sql/stats_ext.sql      |  31 +++++
 8 files changed, 299 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml
index 221a967bfe..d01acc92b8 100644
--- a/doc/src/sgml/ref/psql-ref.sgml
+++ b/doc/src/sgml/ref/psql-ref.sgml
@@ -1918,6 +1918,20 @@ testdb=&gt;
         </para>
         </listitem>
       </varlistentry>
+      
+      <varlistentry>
+        <term><literal>\dX [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
+        <listitem>
+        <para>
+        Lists extended statistics.
+        If <replaceable class="parameter">pattern</replaceable>
+        is specified, only those extended statistics whose names match the
+        pattern are listed.
+        If <literal>+</literal> is appended to the command name, each extended
+        statistics is listed with its size.
+        </para>
+        </listitem>
+      </varlistentry>
 
       <varlistentry>
         <term><literal>\dy[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index 303e7c3ad8..c5ebc1c3f4 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -928,6 +928,9 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd)
 				else
 					success = listExtensions(pattern);
 				break;
+			case 'X':			/* Extended Statistics */
+				success = listExtendedStats(pattern, show_verbose);
+				break;
 			case 'y':			/* Event Triggers */
 				success = listEventTriggers(pattern, show_verbose);
 				break;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index caf97563f4..46f54199fb 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -4392,6 +4392,156 @@ listEventTriggers(const char *pattern, bool verbose)
 	return true;
 }
 
+/*
+ * \dX
+ *
+ * Describes extended statistics.
+ */
+bool
+listExtendedStats(const char *pattern, bool verbose)
+{
+	PQExpBufferData buf;
+	PGresult   *res;
+	printQueryOpt myopt = pset.popt;
+
+	if (pset.sversion < 100000)
+	{
+		char		sverbuf[32];
+
+		pg_log_error("The server (version %s) does not support extended statistics.",
+					 formatPGVersionNumber(pset.sversion, false,
+										   sverbuf, sizeof(sverbuf)));
+		return true;
+	}
+
+	initPQExpBuffer(&buf);
+	printfPQExpBuffer(&buf,
+					  "SELECT \n"
+					  "es.stxnamespace::pg_catalog.regnamespace::text AS \"%s\", \n"
+					  "es.stxname AS \"%s\", \n"
+					  "pg_catalog.format('%%s FROM %%s', \n"
+					  "  (SELECT pg_catalog.string_agg(pg_catalog.quote_ident(a.attname),', ') \n"
+					  "   FROM pg_catalog.unnest(es.stxkeys) s(attnum) \n"
+					  "   JOIN pg_catalog.pg_attribute a \n"
+					  "   ON (es.stxrelid = a.attrelid \n"
+					  "   AND a.attnum = s.attnum \n"
+					  "   AND NOT a.attisdropped)), \n"
+					  "es.stxrelid::regclass) AS \"%s\"",
+					  gettext_noop("Schema"),
+					  gettext_noop("Name"),
+					  gettext_noop("Definition"));
+
+	/*
+	 * Since 12 there are two catalogs - one for the definition, one for the
+	 * data built by ANALYZE. Older releases use a single catalog. Also, 12
+	 * adds the MCV statistics kind.
+	 */
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN es.stxdependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"));
+	}
+	else
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxddependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxdmcv IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'm' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"),
+						  gettext_noop("MCV"));
+	}
+
+	/* In verbose mode, print sizes of the extended statistics objects. */
+	if (verbose)
+	{
+		if (pset.sversion < 120000)
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN es.stxdependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxdependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"));
+		}
+		else
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxddependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxddependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxdmcv IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdmcv)::bigint) \n"
+							  "     WHEN 'm' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"),
+							  gettext_noop("MCV_size"));
+		}
+	}
+
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+	else
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "LEFT JOIN pg_catalog.pg_statistic_ext_data esd \n"
+							 "ON es.oid = esd.stxoid \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+
+	processSQLNamePattern(pset.db, &buf, pattern, false,
+						  false, NULL,
+						  "stxname", NULL,
+						  NULL);
+
+	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
+
+	res = PSQLexec(buf.data);
+	termPQExpBuffer(&buf);
+	if (!res)
+		return false;
+
+	myopt.nullPrint = NULL;
+	myopt.title = _("List of extended statistics");
+	myopt.translate_header = true;
+
+	printQuery(res, &myopt, pset.queryFout, false, pset.logfile);
+
+	PQclear(res);
+	return true;
+}
+
 /*
  * \dC
  *
diff --git a/src/bin/psql/describe.h b/src/bin/psql/describe.h
index 6044e3a082..867e57d851 100644
--- a/src/bin/psql/describe.h
+++ b/src/bin/psql/describe.h
@@ -102,6 +102,9 @@ extern bool listExtensions(const char *pattern);
 /* \dx+ */
 extern bool listExtensionContents(const char *pattern);
 
+/* \dX */
+extern bool listExtendedStats(const char *pattern, bool verbose);
+
 /* \dy */
 extern bool listEventTriggers(const char *pattern, bool verbose);
 
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 9ec1c4e810..e42bc8c54e 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -267,6 +267,7 @@ slashUsage(unsigned short int pager)
 	fprintf(output, _("  \\du[S+] [PATTERN]      list roles\n"));
 	fprintf(output, _("  \\dv[S+] [PATTERN]      list views\n"));
 	fprintf(output, _("  \\dx[+]  [PATTERN]      list extensions\n"));
+	fprintf(output, _("  \\dX[+]  [PATTERN]      list extended statistics\n"));
 	fprintf(output, _("  \\dy     [PATTERN]      list event triggers\n"));
 	fprintf(output, _("  \\l[+]   [PATTERN]      list databases\n"));
 	fprintf(output, _("  \\sf[+]  FUNCNAME       show a function's definition\n"));
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9dcab0d2fa..611f1efb15 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1500,7 +1500,7 @@ psql_completion(const char *text, int start, int end)
 		"\\dF", "\\dFd", "\\dFp", "\\dFt", "\\dg", "\\di", "\\dl", "\\dL",
 		"\\dm", "\\dn", "\\do", "\\dO", "\\dp", "\\dP", "\\dPi", "\\dPt",
 		"\\drds", "\\dRs", "\\dRp", "\\ds", "\\dS",
-		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dy",
+		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dX", "\\dy",
 		"\\e", "\\echo", "\\ef", "\\elif", "\\else", "\\encoding",
 		"\\endif", "\\errverbose", "\\ev",
 		"\\f",
@@ -3910,6 +3910,8 @@ psql_completion(const char *text, int start, int end)
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_views, NULL);
 	else if (TailMatchesCS("\\dx*"))
 		COMPLETE_WITH_QUERY(Query_for_list_of_extensions);
+	else if (TailMatchesCS("\\dX*"))
+		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_statistics, NULL);
 	else if (TailMatchesCS("\\dm*"))
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_matviews, NULL);
 	else if (TailMatchesCS("\\dE*"))
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 7bfeaf85f0..8c8a0afcf6 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -1725,6 +1725,100 @@ INSERT INTO tststats.priv_test_tbl
 CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
   FROM tststats.priv_test_tbl;
 ANALYZE tststats.priv_test_tbl;
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+\dX
+                                          List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   
+----------+------------------------+--------------------------------------+-----------+--------------+---------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built
+(12 rows)
+
+\dX stts_?
+                       List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   
+--------+--------+-------------------+-----------+--------------+---------
+ public | stts_1 | a, b FROM stts_t1 | built     |              | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined
+(4 rows)
+
+\dX *stts_hoge
+                               List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   
+--------+-----------+-------------------------------+-----------+--------------+---------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined
+(1 row)
+
+\dX+
+                                                                   List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+----------+------------------------+--------------------------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        |         |                | 106 bytes         | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built   |                |                   | 24 kB
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built   |                |                   | 386 bytes
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built   |                |                   | 294 bytes
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              |         | 13 bytes       |                   | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined |                | 0 bytes           | 0 bytes
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built   |                |                   | 686 bytes
+(12 rows)
+
+\dX+ stts_?
+                                                List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+--------+--------+-------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public | stts_1 | a, b FROM stts_t1 | built     |              |         | 13 bytes       |                   | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(4 rows)
+
+\dX+ *stts_hoge
+                                                       List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+--------+-----------+-------------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(1 row)
+
+\dX+ stts_s2.stts_yama
+                                                    List of extended statistics
+ Schema  |   Name    |       Definition        | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+---------+-----------+-------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ stts_s2 | stts_yama | col1, col3 FROM stts_t3 |           | defined      | defined |                | 0 bytes           | 0 bytes
+(1 row)
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 7912e733ae..db6e3e1ba3 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -912,6 +912,37 @@ CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
 
 ANALYZE tststats.priv_test_tbl;
 
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+
+\dX
+\dX stts_?
+\dX *stts_hoge
+\dX+
+\dX+ stts_?
+\dX+ *stts_hoge
+\dX+ stts_s2.stts_yama
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
+
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
-- 
2.26.2


--------------B0278A9E952F6668D2DBEBA1
Content-Type: text/x-patch; charset=UTF-8;
 name="0002-fixup-rename-defined-to-requested-20210109.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="0002-fixup-rename-defined-to-requested-20210109.patch"



^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* [PATCH 1/2] psql \dX: list extended statistics objects
@ 2021-01-07 05:28 Tatsuro Yamada <[email protected]>
  0 siblings, 0 replies; 30+ messages in thread

From: Tatsuro Yamada @ 2021-01-07 05:28 UTC (permalink / raw)

The new command lists extended statistics objects, possibly with their
sizes. All past releases with extended statistics are supported.

Author: Tatsuro Yamada
Reviewed-by: Julien Rouhaud, Alvaro Herrera, Tomas Vondra
Discussion: https://postgr.es/m/c027a541-5856-75a5-0868-341301e1624b%40nttcom.co.jp_1
---
 doc/src/sgml/ref/psql-ref.sgml          |  14 +++
 src/bin/psql/command.c                  |   3 +
 src/bin/psql/describe.c                 | 150 ++++++++++++++++++++++++
 src/bin/psql/describe.h                 |   3 +
 src/bin/psql/help.c                     |   1 +
 src/bin/psql/tab-complete.c             |   4 +-
 src/test/regress/expected/stats_ext.out |  94 +++++++++++++++
 src/test/regress/sql/stats_ext.sql      |  31 +++++
 8 files changed, 299 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml
index 221a967bfe..d01acc92b8 100644
--- a/doc/src/sgml/ref/psql-ref.sgml
+++ b/doc/src/sgml/ref/psql-ref.sgml
@@ -1918,6 +1918,20 @@ testdb=&gt;
         </para>
         </listitem>
       </varlistentry>
+      
+      <varlistentry>
+        <term><literal>\dX [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
+        <listitem>
+        <para>
+        Lists extended statistics.
+        If <replaceable class="parameter">pattern</replaceable>
+        is specified, only those extended statistics whose names match the
+        pattern are listed.
+        If <literal>+</literal> is appended to the command name, each extended
+        statistics is listed with its size.
+        </para>
+        </listitem>
+      </varlistentry>
 
       <varlistentry>
         <term><literal>\dy[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index 303e7c3ad8..c5ebc1c3f4 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -928,6 +928,9 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd)
 				else
 					success = listExtensions(pattern);
 				break;
+			case 'X':			/* Extended Statistics */
+				success = listExtendedStats(pattern, show_verbose);
+				break;
 			case 'y':			/* Event Triggers */
 				success = listEventTriggers(pattern, show_verbose);
 				break;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index caf97563f4..46f54199fb 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -4392,6 +4392,156 @@ listEventTriggers(const char *pattern, bool verbose)
 	return true;
 }
 
+/*
+ * \dX
+ *
+ * Describes extended statistics.
+ */
+bool
+listExtendedStats(const char *pattern, bool verbose)
+{
+	PQExpBufferData buf;
+	PGresult   *res;
+	printQueryOpt myopt = pset.popt;
+
+	if (pset.sversion < 100000)
+	{
+		char		sverbuf[32];
+
+		pg_log_error("The server (version %s) does not support extended statistics.",
+					 formatPGVersionNumber(pset.sversion, false,
+										   sverbuf, sizeof(sverbuf)));
+		return true;
+	}
+
+	initPQExpBuffer(&buf);
+	printfPQExpBuffer(&buf,
+					  "SELECT \n"
+					  "es.stxnamespace::pg_catalog.regnamespace::text AS \"%s\", \n"
+					  "es.stxname AS \"%s\", \n"
+					  "pg_catalog.format('%%s FROM %%s', \n"
+					  "  (SELECT pg_catalog.string_agg(pg_catalog.quote_ident(a.attname),', ') \n"
+					  "   FROM pg_catalog.unnest(es.stxkeys) s(attnum) \n"
+					  "   JOIN pg_catalog.pg_attribute a \n"
+					  "   ON (es.stxrelid = a.attrelid \n"
+					  "   AND a.attnum = s.attnum \n"
+					  "   AND NOT a.attisdropped)), \n"
+					  "es.stxrelid::regclass) AS \"%s\"",
+					  gettext_noop("Schema"),
+					  gettext_noop("Name"),
+					  gettext_noop("Definition"));
+
+	/*
+	 * Since 12 there are two catalogs - one for the definition, one for the
+	 * data built by ANALYZE. Older releases use a single catalog. Also, 12
+	 * adds the MCV statistics kind.
+	 */
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN es.stxdependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"));
+	}
+	else
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxddependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxdmcv IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'm' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"),
+						  gettext_noop("MCV"));
+	}
+
+	/* In verbose mode, print sizes of the extended statistics objects. */
+	if (verbose)
+	{
+		if (pset.sversion < 120000)
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN es.stxdependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxdependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"));
+		}
+		else
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxddependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxddependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxdmcv IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdmcv)::bigint) \n"
+							  "     WHEN 'm' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"),
+							  gettext_noop("MCV_size"));
+		}
+	}
+
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+	else
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "LEFT JOIN pg_catalog.pg_statistic_ext_data esd \n"
+							 "ON es.oid = esd.stxoid \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+
+	processSQLNamePattern(pset.db, &buf, pattern, false,
+						  false, NULL,
+						  "stxname", NULL,
+						  NULL);
+
+	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
+
+	res = PSQLexec(buf.data);
+	termPQExpBuffer(&buf);
+	if (!res)
+		return false;
+
+	myopt.nullPrint = NULL;
+	myopt.title = _("List of extended statistics");
+	myopt.translate_header = true;
+
+	printQuery(res, &myopt, pset.queryFout, false, pset.logfile);
+
+	PQclear(res);
+	return true;
+}
+
 /*
  * \dC
  *
diff --git a/src/bin/psql/describe.h b/src/bin/psql/describe.h
index 6044e3a082..867e57d851 100644
--- a/src/bin/psql/describe.h
+++ b/src/bin/psql/describe.h
@@ -102,6 +102,9 @@ extern bool listExtensions(const char *pattern);
 /* \dx+ */
 extern bool listExtensionContents(const char *pattern);
 
+/* \dX */
+extern bool listExtendedStats(const char *pattern, bool verbose);
+
 /* \dy */
 extern bool listEventTriggers(const char *pattern, bool verbose);
 
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 9ec1c4e810..e42bc8c54e 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -267,6 +267,7 @@ slashUsage(unsigned short int pager)
 	fprintf(output, _("  \\du[S+] [PATTERN]      list roles\n"));
 	fprintf(output, _("  \\dv[S+] [PATTERN]      list views\n"));
 	fprintf(output, _("  \\dx[+]  [PATTERN]      list extensions\n"));
+	fprintf(output, _("  \\dX[+]  [PATTERN]      list extended statistics\n"));
 	fprintf(output, _("  \\dy     [PATTERN]      list event triggers\n"));
 	fprintf(output, _("  \\l[+]   [PATTERN]      list databases\n"));
 	fprintf(output, _("  \\sf[+]  FUNCNAME       show a function's definition\n"));
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9dcab0d2fa..611f1efb15 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1500,7 +1500,7 @@ psql_completion(const char *text, int start, int end)
 		"\\dF", "\\dFd", "\\dFp", "\\dFt", "\\dg", "\\di", "\\dl", "\\dL",
 		"\\dm", "\\dn", "\\do", "\\dO", "\\dp", "\\dP", "\\dPi", "\\dPt",
 		"\\drds", "\\dRs", "\\dRp", "\\ds", "\\dS",
-		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dy",
+		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dX", "\\dy",
 		"\\e", "\\echo", "\\ef", "\\elif", "\\else", "\\encoding",
 		"\\endif", "\\errverbose", "\\ev",
 		"\\f",
@@ -3910,6 +3910,8 @@ psql_completion(const char *text, int start, int end)
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_views, NULL);
 	else if (TailMatchesCS("\\dx*"))
 		COMPLETE_WITH_QUERY(Query_for_list_of_extensions);
+	else if (TailMatchesCS("\\dX*"))
+		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_statistics, NULL);
 	else if (TailMatchesCS("\\dm*"))
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_matviews, NULL);
 	else if (TailMatchesCS("\\dE*"))
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 7bfeaf85f0..8c8a0afcf6 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -1725,6 +1725,100 @@ INSERT INTO tststats.priv_test_tbl
 CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
   FROM tststats.priv_test_tbl;
 ANALYZE tststats.priv_test_tbl;
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+\dX
+                                          List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   
+----------+------------------------+--------------------------------------+-----------+--------------+---------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built
+(12 rows)
+
+\dX stts_?
+                       List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   
+--------+--------+-------------------+-----------+--------------+---------
+ public | stts_1 | a, b FROM stts_t1 | built     |              | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined
+(4 rows)
+
+\dX *stts_hoge
+                               List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   
+--------+-----------+-------------------------------+-----------+--------------+---------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined
+(1 row)
+
+\dX+
+                                                                   List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+----------+------------------------+--------------------------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        |         |                | 106 bytes         | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built   |                |                   | 24 kB
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built   |                |                   | 386 bytes
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built   |                |                   | 294 bytes
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              |         | 13 bytes       |                   | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined |                | 0 bytes           | 0 bytes
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built   |                |                   | 686 bytes
+(12 rows)
+
+\dX+ stts_?
+                                                List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+--------+--------+-------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public | stts_1 | a, b FROM stts_t1 | built     |              |         | 13 bytes       |                   | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(4 rows)
+
+\dX+ *stts_hoge
+                                                       List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+--------+-----------+-------------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(1 row)
+
+\dX+ stts_s2.stts_yama
+                                                    List of extended statistics
+ Schema  |   Name    |       Definition        | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+---------+-----------+-------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ stts_s2 | stts_yama | col1, col3 FROM stts_t3 |           | defined      | defined |                | 0 bytes           | 0 bytes
+(1 row)
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 7912e733ae..db6e3e1ba3 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -912,6 +912,37 @@ CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
 
 ANALYZE tststats.priv_test_tbl;
 
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+
+\dX
+\dX stts_?
+\dX *stts_hoge
+\dX+
+\dX+ stts_?
+\dX+ *stts_hoge
+\dX+ stts_s2.stts_yama
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
+
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
-- 
2.26.2


--------------B0278A9E952F6668D2DBEBA1
Content-Type: text/x-patch; charset=UTF-8;
 name="0002-fixup-rename-defined-to-requested-20210109.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="0002-fixup-rename-defined-to-requested-20210109.patch"



^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* [PATCH 1/2] psql \dX: list extended statistics objects
@ 2021-01-07 05:28 Tatsuro Yamada <[email protected]>
  0 siblings, 0 replies; 30+ messages in thread

From: Tatsuro Yamada @ 2021-01-07 05:28 UTC (permalink / raw)

The new command lists extended statistics objects, possibly with their
sizes. All past releases with extended statistics are supported.

Author: Tatsuro Yamada
Reviewed-by: Julien Rouhaud, Alvaro Herrera, Tomas Vondra
Discussion: https://postgr.es/m/c027a541-5856-75a5-0868-341301e1624b%40nttcom.co.jp_1
---
 doc/src/sgml/ref/psql-ref.sgml          |  14 +++
 src/bin/psql/command.c                  |   3 +
 src/bin/psql/describe.c                 | 150 ++++++++++++++++++++++++
 src/bin/psql/describe.h                 |   3 +
 src/bin/psql/help.c                     |   1 +
 src/bin/psql/tab-complete.c             |   4 +-
 src/test/regress/expected/stats_ext.out |  94 +++++++++++++++
 src/test/regress/sql/stats_ext.sql      |  31 +++++
 8 files changed, 299 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml
index 221a967bfe..d01acc92b8 100644
--- a/doc/src/sgml/ref/psql-ref.sgml
+++ b/doc/src/sgml/ref/psql-ref.sgml
@@ -1918,6 +1918,20 @@ testdb=&gt;
         </para>
         </listitem>
       </varlistentry>
+      
+      <varlistentry>
+        <term><literal>\dX [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
+        <listitem>
+        <para>
+        Lists extended statistics.
+        If <replaceable class="parameter">pattern</replaceable>
+        is specified, only those extended statistics whose names match the
+        pattern are listed.
+        If <literal>+</literal> is appended to the command name, each extended
+        statistics is listed with its size.
+        </para>
+        </listitem>
+      </varlistentry>
 
       <varlistentry>
         <term><literal>\dy[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index 303e7c3ad8..c5ebc1c3f4 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -928,6 +928,9 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd)
 				else
 					success = listExtensions(pattern);
 				break;
+			case 'X':			/* Extended Statistics */
+				success = listExtendedStats(pattern, show_verbose);
+				break;
 			case 'y':			/* Event Triggers */
 				success = listEventTriggers(pattern, show_verbose);
 				break;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index caf97563f4..46f54199fb 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -4392,6 +4392,156 @@ listEventTriggers(const char *pattern, bool verbose)
 	return true;
 }
 
+/*
+ * \dX
+ *
+ * Describes extended statistics.
+ */
+bool
+listExtendedStats(const char *pattern, bool verbose)
+{
+	PQExpBufferData buf;
+	PGresult   *res;
+	printQueryOpt myopt = pset.popt;
+
+	if (pset.sversion < 100000)
+	{
+		char		sverbuf[32];
+
+		pg_log_error("The server (version %s) does not support extended statistics.",
+					 formatPGVersionNumber(pset.sversion, false,
+										   sverbuf, sizeof(sverbuf)));
+		return true;
+	}
+
+	initPQExpBuffer(&buf);
+	printfPQExpBuffer(&buf,
+					  "SELECT \n"
+					  "es.stxnamespace::pg_catalog.regnamespace::text AS \"%s\", \n"
+					  "es.stxname AS \"%s\", \n"
+					  "pg_catalog.format('%%s FROM %%s', \n"
+					  "  (SELECT pg_catalog.string_agg(pg_catalog.quote_ident(a.attname),', ') \n"
+					  "   FROM pg_catalog.unnest(es.stxkeys) s(attnum) \n"
+					  "   JOIN pg_catalog.pg_attribute a \n"
+					  "   ON (es.stxrelid = a.attrelid \n"
+					  "   AND a.attnum = s.attnum \n"
+					  "   AND NOT a.attisdropped)), \n"
+					  "es.stxrelid::regclass) AS \"%s\"",
+					  gettext_noop("Schema"),
+					  gettext_noop("Name"),
+					  gettext_noop("Definition"));
+
+	/*
+	 * Since 12 there are two catalogs - one for the definition, one for the
+	 * data built by ANALYZE. Older releases use a single catalog. Also, 12
+	 * adds the MCV statistics kind.
+	 */
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN es.stxdependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"));
+	}
+	else
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxddependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxdmcv IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'm' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"),
+						  gettext_noop("MCV"));
+	}
+
+	/* In verbose mode, print sizes of the extended statistics objects. */
+	if (verbose)
+	{
+		if (pset.sversion < 120000)
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN es.stxdependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxdependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"));
+		}
+		else
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxddependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxddependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxdmcv IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdmcv)::bigint) \n"
+							  "     WHEN 'm' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"),
+							  gettext_noop("MCV_size"));
+		}
+	}
+
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+	else
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "LEFT JOIN pg_catalog.pg_statistic_ext_data esd \n"
+							 "ON es.oid = esd.stxoid \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+
+	processSQLNamePattern(pset.db, &buf, pattern, false,
+						  false, NULL,
+						  "stxname", NULL,
+						  NULL);
+
+	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
+
+	res = PSQLexec(buf.data);
+	termPQExpBuffer(&buf);
+	if (!res)
+		return false;
+
+	myopt.nullPrint = NULL;
+	myopt.title = _("List of extended statistics");
+	myopt.translate_header = true;
+
+	printQuery(res, &myopt, pset.queryFout, false, pset.logfile);
+
+	PQclear(res);
+	return true;
+}
+
 /*
  * \dC
  *
diff --git a/src/bin/psql/describe.h b/src/bin/psql/describe.h
index 6044e3a082..867e57d851 100644
--- a/src/bin/psql/describe.h
+++ b/src/bin/psql/describe.h
@@ -102,6 +102,9 @@ extern bool listExtensions(const char *pattern);
 /* \dx+ */
 extern bool listExtensionContents(const char *pattern);
 
+/* \dX */
+extern bool listExtendedStats(const char *pattern, bool verbose);
+
 /* \dy */
 extern bool listEventTriggers(const char *pattern, bool verbose);
 
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 9ec1c4e810..e42bc8c54e 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -267,6 +267,7 @@ slashUsage(unsigned short int pager)
 	fprintf(output, _("  \\du[S+] [PATTERN]      list roles\n"));
 	fprintf(output, _("  \\dv[S+] [PATTERN]      list views\n"));
 	fprintf(output, _("  \\dx[+]  [PATTERN]      list extensions\n"));
+	fprintf(output, _("  \\dX[+]  [PATTERN]      list extended statistics\n"));
 	fprintf(output, _("  \\dy     [PATTERN]      list event triggers\n"));
 	fprintf(output, _("  \\l[+]   [PATTERN]      list databases\n"));
 	fprintf(output, _("  \\sf[+]  FUNCNAME       show a function's definition\n"));
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9dcab0d2fa..611f1efb15 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1500,7 +1500,7 @@ psql_completion(const char *text, int start, int end)
 		"\\dF", "\\dFd", "\\dFp", "\\dFt", "\\dg", "\\di", "\\dl", "\\dL",
 		"\\dm", "\\dn", "\\do", "\\dO", "\\dp", "\\dP", "\\dPi", "\\dPt",
 		"\\drds", "\\dRs", "\\dRp", "\\ds", "\\dS",
-		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dy",
+		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dX", "\\dy",
 		"\\e", "\\echo", "\\ef", "\\elif", "\\else", "\\encoding",
 		"\\endif", "\\errverbose", "\\ev",
 		"\\f",
@@ -3910,6 +3910,8 @@ psql_completion(const char *text, int start, int end)
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_views, NULL);
 	else if (TailMatchesCS("\\dx*"))
 		COMPLETE_WITH_QUERY(Query_for_list_of_extensions);
+	else if (TailMatchesCS("\\dX*"))
+		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_statistics, NULL);
 	else if (TailMatchesCS("\\dm*"))
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_matviews, NULL);
 	else if (TailMatchesCS("\\dE*"))
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 7bfeaf85f0..8c8a0afcf6 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -1725,6 +1725,100 @@ INSERT INTO tststats.priv_test_tbl
 CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
   FROM tststats.priv_test_tbl;
 ANALYZE tststats.priv_test_tbl;
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+\dX
+                                          List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   
+----------+------------------------+--------------------------------------+-----------+--------------+---------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built
+(12 rows)
+
+\dX stts_?
+                       List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   
+--------+--------+-------------------+-----------+--------------+---------
+ public | stts_1 | a, b FROM stts_t1 | built     |              | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined
+(4 rows)
+
+\dX *stts_hoge
+                               List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   
+--------+-----------+-------------------------------+-----------+--------------+---------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined
+(1 row)
+
+\dX+
+                                                                   List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+----------+------------------------+--------------------------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        |         |                | 106 bytes         | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built   |                |                   | 24 kB
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built   |                |                   | 386 bytes
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built   |                |                   | 294 bytes
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              |         | 13 bytes       |                   | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined |                | 0 bytes           | 0 bytes
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built   |                |                   | 686 bytes
+(12 rows)
+
+\dX+ stts_?
+                                                List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+--------+--------+-------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public | stts_1 | a, b FROM stts_t1 | built     |              |         | 13 bytes       |                   | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(4 rows)
+
+\dX+ *stts_hoge
+                                                       List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+--------+-----------+-------------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(1 row)
+
+\dX+ stts_s2.stts_yama
+                                                    List of extended statistics
+ Schema  |   Name    |       Definition        | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+---------+-----------+-------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ stts_s2 | stts_yama | col1, col3 FROM stts_t3 |           | defined      | defined |                | 0 bytes           | 0 bytes
+(1 row)
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 7912e733ae..db6e3e1ba3 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -912,6 +912,37 @@ CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
 
 ANALYZE tststats.priv_test_tbl;
 
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+
+\dX
+\dX stts_?
+\dX *stts_hoge
+\dX+
+\dX+ stts_?
+\dX+ *stts_hoge
+\dX+ stts_s2.stts_yama
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
+
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
-- 
2.26.2


--------------B0278A9E952F6668D2DBEBA1
Content-Type: text/x-patch; charset=UTF-8;
 name="0002-fixup-rename-defined-to-requested-20210109.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="0002-fixup-rename-defined-to-requested-20210109.patch"



^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* [PATCH 1/2] psql \dX: list extended statistics objects
@ 2021-01-07 05:28 Tatsuro Yamada <[email protected]>
  0 siblings, 0 replies; 30+ messages in thread

From: Tatsuro Yamada @ 2021-01-07 05:28 UTC (permalink / raw)

The new command lists extended statistics objects, possibly with their
sizes. All past releases with extended statistics are supported.

Author: Tatsuro Yamada
Reviewed-by: Julien Rouhaud, Alvaro Herrera, Tomas Vondra
Discussion: https://postgr.es/m/c027a541-5856-75a5-0868-341301e1624b%40nttcom.co.jp_1
---
 doc/src/sgml/ref/psql-ref.sgml          |  14 +++
 src/bin/psql/command.c                  |   3 +
 src/bin/psql/describe.c                 | 150 ++++++++++++++++++++++++
 src/bin/psql/describe.h                 |   3 +
 src/bin/psql/help.c                     |   1 +
 src/bin/psql/tab-complete.c             |   4 +-
 src/test/regress/expected/stats_ext.out |  94 +++++++++++++++
 src/test/regress/sql/stats_ext.sql      |  31 +++++
 8 files changed, 299 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml
index 221a967bfe..d01acc92b8 100644
--- a/doc/src/sgml/ref/psql-ref.sgml
+++ b/doc/src/sgml/ref/psql-ref.sgml
@@ -1918,6 +1918,20 @@ testdb=&gt;
         </para>
         </listitem>
       </varlistentry>
+      
+      <varlistentry>
+        <term><literal>\dX [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
+        <listitem>
+        <para>
+        Lists extended statistics.
+        If <replaceable class="parameter">pattern</replaceable>
+        is specified, only those extended statistics whose names match the
+        pattern are listed.
+        If <literal>+</literal> is appended to the command name, each extended
+        statistics is listed with its size.
+        </para>
+        </listitem>
+      </varlistentry>
 
       <varlistentry>
         <term><literal>\dy[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index 303e7c3ad8..c5ebc1c3f4 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -928,6 +928,9 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd)
 				else
 					success = listExtensions(pattern);
 				break;
+			case 'X':			/* Extended Statistics */
+				success = listExtendedStats(pattern, show_verbose);
+				break;
 			case 'y':			/* Event Triggers */
 				success = listEventTriggers(pattern, show_verbose);
 				break;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index caf97563f4..46f54199fb 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -4392,6 +4392,156 @@ listEventTriggers(const char *pattern, bool verbose)
 	return true;
 }
 
+/*
+ * \dX
+ *
+ * Describes extended statistics.
+ */
+bool
+listExtendedStats(const char *pattern, bool verbose)
+{
+	PQExpBufferData buf;
+	PGresult   *res;
+	printQueryOpt myopt = pset.popt;
+
+	if (pset.sversion < 100000)
+	{
+		char		sverbuf[32];
+
+		pg_log_error("The server (version %s) does not support extended statistics.",
+					 formatPGVersionNumber(pset.sversion, false,
+										   sverbuf, sizeof(sverbuf)));
+		return true;
+	}
+
+	initPQExpBuffer(&buf);
+	printfPQExpBuffer(&buf,
+					  "SELECT \n"
+					  "es.stxnamespace::pg_catalog.regnamespace::text AS \"%s\", \n"
+					  "es.stxname AS \"%s\", \n"
+					  "pg_catalog.format('%%s FROM %%s', \n"
+					  "  (SELECT pg_catalog.string_agg(pg_catalog.quote_ident(a.attname),', ') \n"
+					  "   FROM pg_catalog.unnest(es.stxkeys) s(attnum) \n"
+					  "   JOIN pg_catalog.pg_attribute a \n"
+					  "   ON (es.stxrelid = a.attrelid \n"
+					  "   AND a.attnum = s.attnum \n"
+					  "   AND NOT a.attisdropped)), \n"
+					  "es.stxrelid::regclass) AS \"%s\"",
+					  gettext_noop("Schema"),
+					  gettext_noop("Name"),
+					  gettext_noop("Definition"));
+
+	/*
+	 * Since 12 there are two catalogs - one for the definition, one for the
+	 * data built by ANALYZE. Older releases use a single catalog. Also, 12
+	 * adds the MCV statistics kind.
+	 */
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN es.stxdependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"));
+	}
+	else
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxddependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxdmcv IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'm' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"),
+						  gettext_noop("MCV"));
+	}
+
+	/* In verbose mode, print sizes of the extended statistics objects. */
+	if (verbose)
+	{
+		if (pset.sversion < 120000)
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN es.stxdependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxdependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"));
+		}
+		else
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxddependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxddependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxdmcv IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdmcv)::bigint) \n"
+							  "     WHEN 'm' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"),
+							  gettext_noop("MCV_size"));
+		}
+	}
+
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+	else
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "LEFT JOIN pg_catalog.pg_statistic_ext_data esd \n"
+							 "ON es.oid = esd.stxoid \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+
+	processSQLNamePattern(pset.db, &buf, pattern, false,
+						  false, NULL,
+						  "stxname", NULL,
+						  NULL);
+
+	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
+
+	res = PSQLexec(buf.data);
+	termPQExpBuffer(&buf);
+	if (!res)
+		return false;
+
+	myopt.nullPrint = NULL;
+	myopt.title = _("List of extended statistics");
+	myopt.translate_header = true;
+
+	printQuery(res, &myopt, pset.queryFout, false, pset.logfile);
+
+	PQclear(res);
+	return true;
+}
+
 /*
  * \dC
  *
diff --git a/src/bin/psql/describe.h b/src/bin/psql/describe.h
index 6044e3a082..867e57d851 100644
--- a/src/bin/psql/describe.h
+++ b/src/bin/psql/describe.h
@@ -102,6 +102,9 @@ extern bool listExtensions(const char *pattern);
 /* \dx+ */
 extern bool listExtensionContents(const char *pattern);
 
+/* \dX */
+extern bool listExtendedStats(const char *pattern, bool verbose);
+
 /* \dy */
 extern bool listEventTriggers(const char *pattern, bool verbose);
 
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 9ec1c4e810..e42bc8c54e 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -267,6 +267,7 @@ slashUsage(unsigned short int pager)
 	fprintf(output, _("  \\du[S+] [PATTERN]      list roles\n"));
 	fprintf(output, _("  \\dv[S+] [PATTERN]      list views\n"));
 	fprintf(output, _("  \\dx[+]  [PATTERN]      list extensions\n"));
+	fprintf(output, _("  \\dX[+]  [PATTERN]      list extended statistics\n"));
 	fprintf(output, _("  \\dy     [PATTERN]      list event triggers\n"));
 	fprintf(output, _("  \\l[+]   [PATTERN]      list databases\n"));
 	fprintf(output, _("  \\sf[+]  FUNCNAME       show a function's definition\n"));
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9dcab0d2fa..611f1efb15 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1500,7 +1500,7 @@ psql_completion(const char *text, int start, int end)
 		"\\dF", "\\dFd", "\\dFp", "\\dFt", "\\dg", "\\di", "\\dl", "\\dL",
 		"\\dm", "\\dn", "\\do", "\\dO", "\\dp", "\\dP", "\\dPi", "\\dPt",
 		"\\drds", "\\dRs", "\\dRp", "\\ds", "\\dS",
-		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dy",
+		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dX", "\\dy",
 		"\\e", "\\echo", "\\ef", "\\elif", "\\else", "\\encoding",
 		"\\endif", "\\errverbose", "\\ev",
 		"\\f",
@@ -3910,6 +3910,8 @@ psql_completion(const char *text, int start, int end)
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_views, NULL);
 	else if (TailMatchesCS("\\dx*"))
 		COMPLETE_WITH_QUERY(Query_for_list_of_extensions);
+	else if (TailMatchesCS("\\dX*"))
+		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_statistics, NULL);
 	else if (TailMatchesCS("\\dm*"))
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_matviews, NULL);
 	else if (TailMatchesCS("\\dE*"))
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 7bfeaf85f0..8c8a0afcf6 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -1725,6 +1725,100 @@ INSERT INTO tststats.priv_test_tbl
 CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
   FROM tststats.priv_test_tbl;
 ANALYZE tststats.priv_test_tbl;
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+\dX
+                                          List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   
+----------+------------------------+--------------------------------------+-----------+--------------+---------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built
+(12 rows)
+
+\dX stts_?
+                       List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   
+--------+--------+-------------------+-----------+--------------+---------
+ public | stts_1 | a, b FROM stts_t1 | built     |              | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined
+(4 rows)
+
+\dX *stts_hoge
+                               List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   
+--------+-----------+-------------------------------+-----------+--------------+---------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined
+(1 row)
+
+\dX+
+                                                                   List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+----------+------------------------+--------------------------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        |         |                | 106 bytes         | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built   |                |                   | 24 kB
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built   |                |                   | 386 bytes
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built   |                |                   | 294 bytes
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              |         | 13 bytes       |                   | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined |                | 0 bytes           | 0 bytes
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built   |                |                   | 686 bytes
+(12 rows)
+
+\dX+ stts_?
+                                                List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+--------+--------+-------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public | stts_1 | a, b FROM stts_t1 | built     |              |         | 13 bytes       |                   | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(4 rows)
+
+\dX+ *stts_hoge
+                                                       List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+--------+-----------+-------------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(1 row)
+
+\dX+ stts_s2.stts_yama
+                                                    List of extended statistics
+ Schema  |   Name    |       Definition        | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+---------+-----------+-------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ stts_s2 | stts_yama | col1, col3 FROM stts_t3 |           | defined      | defined |                | 0 bytes           | 0 bytes
+(1 row)
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 7912e733ae..db6e3e1ba3 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -912,6 +912,37 @@ CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
 
 ANALYZE tststats.priv_test_tbl;
 
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+
+\dX
+\dX stts_?
+\dX *stts_hoge
+\dX+
+\dX+ stts_?
+\dX+ *stts_hoge
+\dX+ stts_s2.stts_yama
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
+
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
-- 
2.26.2


--------------B0278A9E952F6668D2DBEBA1
Content-Type: text/x-patch; charset=UTF-8;
 name="0002-fixup-rename-defined-to-requested-20210109.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="0002-fixup-rename-defined-to-requested-20210109.patch"



^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* [PATCH 1/2] psql \dX: list extended statistics objects
@ 2021-01-07 05:28 Tatsuro Yamada <[email protected]>
  0 siblings, 0 replies; 30+ messages in thread

From: Tatsuro Yamada @ 2021-01-07 05:28 UTC (permalink / raw)

The new command lists extended statistics objects, possibly with their
sizes. All past releases with extended statistics are supported.

Author: Tatsuro Yamada
Reviewed-by: Julien Rouhaud, Alvaro Herrera, Tomas Vondra
Discussion: https://postgr.es/m/c027a541-5856-75a5-0868-341301e1624b%40nttcom.co.jp_1
---
 doc/src/sgml/ref/psql-ref.sgml          |  14 +++
 src/bin/psql/command.c                  |   3 +
 src/bin/psql/describe.c                 | 150 ++++++++++++++++++++++++
 src/bin/psql/describe.h                 |   3 +
 src/bin/psql/help.c                     |   1 +
 src/bin/psql/tab-complete.c             |   4 +-
 src/test/regress/expected/stats_ext.out |  94 +++++++++++++++
 src/test/regress/sql/stats_ext.sql      |  31 +++++
 8 files changed, 299 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml
index 221a967bfe..d01acc92b8 100644
--- a/doc/src/sgml/ref/psql-ref.sgml
+++ b/doc/src/sgml/ref/psql-ref.sgml
@@ -1918,6 +1918,20 @@ testdb=&gt;
         </para>
         </listitem>
       </varlistentry>
+      
+      <varlistentry>
+        <term><literal>\dX [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
+        <listitem>
+        <para>
+        Lists extended statistics.
+        If <replaceable class="parameter">pattern</replaceable>
+        is specified, only those extended statistics whose names match the
+        pattern are listed.
+        If <literal>+</literal> is appended to the command name, each extended
+        statistics is listed with its size.
+        </para>
+        </listitem>
+      </varlistentry>
 
       <varlistentry>
         <term><literal>\dy[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index 303e7c3ad8..c5ebc1c3f4 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -928,6 +928,9 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd)
 				else
 					success = listExtensions(pattern);
 				break;
+			case 'X':			/* Extended Statistics */
+				success = listExtendedStats(pattern, show_verbose);
+				break;
 			case 'y':			/* Event Triggers */
 				success = listEventTriggers(pattern, show_verbose);
 				break;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index caf97563f4..46f54199fb 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -4392,6 +4392,156 @@ listEventTriggers(const char *pattern, bool verbose)
 	return true;
 }
 
+/*
+ * \dX
+ *
+ * Describes extended statistics.
+ */
+bool
+listExtendedStats(const char *pattern, bool verbose)
+{
+	PQExpBufferData buf;
+	PGresult   *res;
+	printQueryOpt myopt = pset.popt;
+
+	if (pset.sversion < 100000)
+	{
+		char		sverbuf[32];
+
+		pg_log_error("The server (version %s) does not support extended statistics.",
+					 formatPGVersionNumber(pset.sversion, false,
+										   sverbuf, sizeof(sverbuf)));
+		return true;
+	}
+
+	initPQExpBuffer(&buf);
+	printfPQExpBuffer(&buf,
+					  "SELECT \n"
+					  "es.stxnamespace::pg_catalog.regnamespace::text AS \"%s\", \n"
+					  "es.stxname AS \"%s\", \n"
+					  "pg_catalog.format('%%s FROM %%s', \n"
+					  "  (SELECT pg_catalog.string_agg(pg_catalog.quote_ident(a.attname),', ') \n"
+					  "   FROM pg_catalog.unnest(es.stxkeys) s(attnum) \n"
+					  "   JOIN pg_catalog.pg_attribute a \n"
+					  "   ON (es.stxrelid = a.attrelid \n"
+					  "   AND a.attnum = s.attnum \n"
+					  "   AND NOT a.attisdropped)), \n"
+					  "es.stxrelid::regclass) AS \"%s\"",
+					  gettext_noop("Schema"),
+					  gettext_noop("Name"),
+					  gettext_noop("Definition"));
+
+	/*
+	 * Since 12 there are two catalogs - one for the definition, one for the
+	 * data built by ANALYZE. Older releases use a single catalog. Also, 12
+	 * adds the MCV statistics kind.
+	 */
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN es.stxdependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"));
+	}
+	else
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxddependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxdmcv IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'm' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"),
+						  gettext_noop("MCV"));
+	}
+
+	/* In verbose mode, print sizes of the extended statistics objects. */
+	if (verbose)
+	{
+		if (pset.sversion < 120000)
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN es.stxdependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxdependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"));
+		}
+		else
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxddependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxddependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxdmcv IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdmcv)::bigint) \n"
+							  "     WHEN 'm' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"),
+							  gettext_noop("MCV_size"));
+		}
+	}
+
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+	else
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "LEFT JOIN pg_catalog.pg_statistic_ext_data esd \n"
+							 "ON es.oid = esd.stxoid \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+
+	processSQLNamePattern(pset.db, &buf, pattern, false,
+						  false, NULL,
+						  "stxname", NULL,
+						  NULL);
+
+	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
+
+	res = PSQLexec(buf.data);
+	termPQExpBuffer(&buf);
+	if (!res)
+		return false;
+
+	myopt.nullPrint = NULL;
+	myopt.title = _("List of extended statistics");
+	myopt.translate_header = true;
+
+	printQuery(res, &myopt, pset.queryFout, false, pset.logfile);
+
+	PQclear(res);
+	return true;
+}
+
 /*
  * \dC
  *
diff --git a/src/bin/psql/describe.h b/src/bin/psql/describe.h
index 6044e3a082..867e57d851 100644
--- a/src/bin/psql/describe.h
+++ b/src/bin/psql/describe.h
@@ -102,6 +102,9 @@ extern bool listExtensions(const char *pattern);
 /* \dx+ */
 extern bool listExtensionContents(const char *pattern);
 
+/* \dX */
+extern bool listExtendedStats(const char *pattern, bool verbose);
+
 /* \dy */
 extern bool listEventTriggers(const char *pattern, bool verbose);
 
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 9ec1c4e810..e42bc8c54e 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -267,6 +267,7 @@ slashUsage(unsigned short int pager)
 	fprintf(output, _("  \\du[S+] [PATTERN]      list roles\n"));
 	fprintf(output, _("  \\dv[S+] [PATTERN]      list views\n"));
 	fprintf(output, _("  \\dx[+]  [PATTERN]      list extensions\n"));
+	fprintf(output, _("  \\dX[+]  [PATTERN]      list extended statistics\n"));
 	fprintf(output, _("  \\dy     [PATTERN]      list event triggers\n"));
 	fprintf(output, _("  \\l[+]   [PATTERN]      list databases\n"));
 	fprintf(output, _("  \\sf[+]  FUNCNAME       show a function's definition\n"));
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9dcab0d2fa..611f1efb15 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1500,7 +1500,7 @@ psql_completion(const char *text, int start, int end)
 		"\\dF", "\\dFd", "\\dFp", "\\dFt", "\\dg", "\\di", "\\dl", "\\dL",
 		"\\dm", "\\dn", "\\do", "\\dO", "\\dp", "\\dP", "\\dPi", "\\dPt",
 		"\\drds", "\\dRs", "\\dRp", "\\ds", "\\dS",
-		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dy",
+		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dX", "\\dy",
 		"\\e", "\\echo", "\\ef", "\\elif", "\\else", "\\encoding",
 		"\\endif", "\\errverbose", "\\ev",
 		"\\f",
@@ -3910,6 +3910,8 @@ psql_completion(const char *text, int start, int end)
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_views, NULL);
 	else if (TailMatchesCS("\\dx*"))
 		COMPLETE_WITH_QUERY(Query_for_list_of_extensions);
+	else if (TailMatchesCS("\\dX*"))
+		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_statistics, NULL);
 	else if (TailMatchesCS("\\dm*"))
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_matviews, NULL);
 	else if (TailMatchesCS("\\dE*"))
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 7bfeaf85f0..8c8a0afcf6 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -1725,6 +1725,100 @@ INSERT INTO tststats.priv_test_tbl
 CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
   FROM tststats.priv_test_tbl;
 ANALYZE tststats.priv_test_tbl;
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+\dX
+                                          List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   
+----------+------------------------+--------------------------------------+-----------+--------------+---------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built
+(12 rows)
+
+\dX stts_?
+                       List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   
+--------+--------+-------------------+-----------+--------------+---------
+ public | stts_1 | a, b FROM stts_t1 | built     |              | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined
+(4 rows)
+
+\dX *stts_hoge
+                               List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   
+--------+-----------+-------------------------------+-----------+--------------+---------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined
+(1 row)
+
+\dX+
+                                                                   List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+----------+------------------------+--------------------------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        |         |                | 106 bytes         | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built   |                |                   | 24 kB
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built   |                |                   | 386 bytes
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built   |                |                   | 294 bytes
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              |         | 13 bytes       |                   | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined |                | 0 bytes           | 0 bytes
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built   |                |                   | 686 bytes
+(12 rows)
+
+\dX+ stts_?
+                                                List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+--------+--------+-------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public | stts_1 | a, b FROM stts_t1 | built     |              |         | 13 bytes       |                   | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(4 rows)
+
+\dX+ *stts_hoge
+                                                       List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+--------+-----------+-------------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(1 row)
+
+\dX+ stts_s2.stts_yama
+                                                    List of extended statistics
+ Schema  |   Name    |       Definition        | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+---------+-----------+-------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ stts_s2 | stts_yama | col1, col3 FROM stts_t3 |           | defined      | defined |                | 0 bytes           | 0 bytes
+(1 row)
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 7912e733ae..db6e3e1ba3 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -912,6 +912,37 @@ CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
 
 ANALYZE tststats.priv_test_tbl;
 
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+
+\dX
+\dX stts_?
+\dX *stts_hoge
+\dX+
+\dX+ stts_?
+\dX+ *stts_hoge
+\dX+ stts_s2.stts_yama
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
+
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
-- 
2.26.2


--------------B0278A9E952F6668D2DBEBA1
Content-Type: text/x-patch; charset=UTF-8;
 name="0002-fixup-rename-defined-to-requested-20210109.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="0002-fixup-rename-defined-to-requested-20210109.patch"



^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* [PATCH 1/2] psql \dX: list extended statistics objects
@ 2021-01-07 05:28 Tatsuro Yamada <[email protected]>
  0 siblings, 0 replies; 30+ messages in thread

From: Tatsuro Yamada @ 2021-01-07 05:28 UTC (permalink / raw)

The new command lists extended statistics objects, possibly with their
sizes. All past releases with extended statistics are supported.

Author: Tatsuro Yamada
Reviewed-by: Julien Rouhaud, Alvaro Herrera, Tomas Vondra
Discussion: https://postgr.es/m/c027a541-5856-75a5-0868-341301e1624b%40nttcom.co.jp_1
---
 doc/src/sgml/ref/psql-ref.sgml          |  14 +++
 src/bin/psql/command.c                  |   3 +
 src/bin/psql/describe.c                 | 150 ++++++++++++++++++++++++
 src/bin/psql/describe.h                 |   3 +
 src/bin/psql/help.c                     |   1 +
 src/bin/psql/tab-complete.c             |   4 +-
 src/test/regress/expected/stats_ext.out |  94 +++++++++++++++
 src/test/regress/sql/stats_ext.sql      |  31 +++++
 8 files changed, 299 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml
index 221a967bfe..d01acc92b8 100644
--- a/doc/src/sgml/ref/psql-ref.sgml
+++ b/doc/src/sgml/ref/psql-ref.sgml
@@ -1918,6 +1918,20 @@ testdb=&gt;
         </para>
         </listitem>
       </varlistentry>
+      
+      <varlistentry>
+        <term><literal>\dX [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
+        <listitem>
+        <para>
+        Lists extended statistics.
+        If <replaceable class="parameter">pattern</replaceable>
+        is specified, only those extended statistics whose names match the
+        pattern are listed.
+        If <literal>+</literal> is appended to the command name, each extended
+        statistics is listed with its size.
+        </para>
+        </listitem>
+      </varlistentry>
 
       <varlistentry>
         <term><literal>\dy[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index 303e7c3ad8..c5ebc1c3f4 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -928,6 +928,9 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd)
 				else
 					success = listExtensions(pattern);
 				break;
+			case 'X':			/* Extended Statistics */
+				success = listExtendedStats(pattern, show_verbose);
+				break;
 			case 'y':			/* Event Triggers */
 				success = listEventTriggers(pattern, show_verbose);
 				break;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index caf97563f4..46f54199fb 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -4392,6 +4392,156 @@ listEventTriggers(const char *pattern, bool verbose)
 	return true;
 }
 
+/*
+ * \dX
+ *
+ * Describes extended statistics.
+ */
+bool
+listExtendedStats(const char *pattern, bool verbose)
+{
+	PQExpBufferData buf;
+	PGresult   *res;
+	printQueryOpt myopt = pset.popt;
+
+	if (pset.sversion < 100000)
+	{
+		char		sverbuf[32];
+
+		pg_log_error("The server (version %s) does not support extended statistics.",
+					 formatPGVersionNumber(pset.sversion, false,
+										   sverbuf, sizeof(sverbuf)));
+		return true;
+	}
+
+	initPQExpBuffer(&buf);
+	printfPQExpBuffer(&buf,
+					  "SELECT \n"
+					  "es.stxnamespace::pg_catalog.regnamespace::text AS \"%s\", \n"
+					  "es.stxname AS \"%s\", \n"
+					  "pg_catalog.format('%%s FROM %%s', \n"
+					  "  (SELECT pg_catalog.string_agg(pg_catalog.quote_ident(a.attname),', ') \n"
+					  "   FROM pg_catalog.unnest(es.stxkeys) s(attnum) \n"
+					  "   JOIN pg_catalog.pg_attribute a \n"
+					  "   ON (es.stxrelid = a.attrelid \n"
+					  "   AND a.attnum = s.attnum \n"
+					  "   AND NOT a.attisdropped)), \n"
+					  "es.stxrelid::regclass) AS \"%s\"",
+					  gettext_noop("Schema"),
+					  gettext_noop("Name"),
+					  gettext_noop("Definition"));
+
+	/*
+	 * Since 12 there are two catalogs - one for the definition, one for the
+	 * data built by ANALYZE. Older releases use a single catalog. Also, 12
+	 * adds the MCV statistics kind.
+	 */
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN es.stxdependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"));
+	}
+	else
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxddependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxdmcv IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'm' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"),
+						  gettext_noop("MCV"));
+	}
+
+	/* In verbose mode, print sizes of the extended statistics objects. */
+	if (verbose)
+	{
+		if (pset.sversion < 120000)
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN es.stxdependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxdependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"));
+		}
+		else
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxddependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxddependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxdmcv IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdmcv)::bigint) \n"
+							  "     WHEN 'm' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"),
+							  gettext_noop("MCV_size"));
+		}
+	}
+
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+	else
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "LEFT JOIN pg_catalog.pg_statistic_ext_data esd \n"
+							 "ON es.oid = esd.stxoid \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+
+	processSQLNamePattern(pset.db, &buf, pattern, false,
+						  false, NULL,
+						  "stxname", NULL,
+						  NULL);
+
+	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
+
+	res = PSQLexec(buf.data);
+	termPQExpBuffer(&buf);
+	if (!res)
+		return false;
+
+	myopt.nullPrint = NULL;
+	myopt.title = _("List of extended statistics");
+	myopt.translate_header = true;
+
+	printQuery(res, &myopt, pset.queryFout, false, pset.logfile);
+
+	PQclear(res);
+	return true;
+}
+
 /*
  * \dC
  *
diff --git a/src/bin/psql/describe.h b/src/bin/psql/describe.h
index 6044e3a082..867e57d851 100644
--- a/src/bin/psql/describe.h
+++ b/src/bin/psql/describe.h
@@ -102,6 +102,9 @@ extern bool listExtensions(const char *pattern);
 /* \dx+ */
 extern bool listExtensionContents(const char *pattern);
 
+/* \dX */
+extern bool listExtendedStats(const char *pattern, bool verbose);
+
 /* \dy */
 extern bool listEventTriggers(const char *pattern, bool verbose);
 
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 9ec1c4e810..e42bc8c54e 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -267,6 +267,7 @@ slashUsage(unsigned short int pager)
 	fprintf(output, _("  \\du[S+] [PATTERN]      list roles\n"));
 	fprintf(output, _("  \\dv[S+] [PATTERN]      list views\n"));
 	fprintf(output, _("  \\dx[+]  [PATTERN]      list extensions\n"));
+	fprintf(output, _("  \\dX[+]  [PATTERN]      list extended statistics\n"));
 	fprintf(output, _("  \\dy     [PATTERN]      list event triggers\n"));
 	fprintf(output, _("  \\l[+]   [PATTERN]      list databases\n"));
 	fprintf(output, _("  \\sf[+]  FUNCNAME       show a function's definition\n"));
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9dcab0d2fa..611f1efb15 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1500,7 +1500,7 @@ psql_completion(const char *text, int start, int end)
 		"\\dF", "\\dFd", "\\dFp", "\\dFt", "\\dg", "\\di", "\\dl", "\\dL",
 		"\\dm", "\\dn", "\\do", "\\dO", "\\dp", "\\dP", "\\dPi", "\\dPt",
 		"\\drds", "\\dRs", "\\dRp", "\\ds", "\\dS",
-		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dy",
+		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dX", "\\dy",
 		"\\e", "\\echo", "\\ef", "\\elif", "\\else", "\\encoding",
 		"\\endif", "\\errverbose", "\\ev",
 		"\\f",
@@ -3910,6 +3910,8 @@ psql_completion(const char *text, int start, int end)
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_views, NULL);
 	else if (TailMatchesCS("\\dx*"))
 		COMPLETE_WITH_QUERY(Query_for_list_of_extensions);
+	else if (TailMatchesCS("\\dX*"))
+		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_statistics, NULL);
 	else if (TailMatchesCS("\\dm*"))
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_matviews, NULL);
 	else if (TailMatchesCS("\\dE*"))
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 7bfeaf85f0..8c8a0afcf6 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -1725,6 +1725,100 @@ INSERT INTO tststats.priv_test_tbl
 CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
   FROM tststats.priv_test_tbl;
 ANALYZE tststats.priv_test_tbl;
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+\dX
+                                          List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   
+----------+------------------------+--------------------------------------+-----------+--------------+---------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built
+(12 rows)
+
+\dX stts_?
+                       List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   
+--------+--------+-------------------+-----------+--------------+---------
+ public | stts_1 | a, b FROM stts_t1 | built     |              | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined
+(4 rows)
+
+\dX *stts_hoge
+                               List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   
+--------+-----------+-------------------------------+-----------+--------------+---------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined
+(1 row)
+
+\dX+
+                                                                   List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+----------+------------------------+--------------------------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        |         |                | 106 bytes         | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built   |                |                   | 24 kB
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built   |                |                   | 386 bytes
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built   |                |                   | 294 bytes
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              |         | 13 bytes       |                   | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined |                | 0 bytes           | 0 bytes
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built   |                |                   | 686 bytes
+(12 rows)
+
+\dX+ stts_?
+                                                List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+--------+--------+-------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public | stts_1 | a, b FROM stts_t1 | built     |              |         | 13 bytes       |                   | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(4 rows)
+
+\dX+ *stts_hoge
+                                                       List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+--------+-----------+-------------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(1 row)
+
+\dX+ stts_s2.stts_yama
+                                                    List of extended statistics
+ Schema  |   Name    |       Definition        | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+---------+-----------+-------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ stts_s2 | stts_yama | col1, col3 FROM stts_t3 |           | defined      | defined |                | 0 bytes           | 0 bytes
+(1 row)
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 7912e733ae..db6e3e1ba3 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -912,6 +912,37 @@ CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
 
 ANALYZE tststats.priv_test_tbl;
 
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+
+\dX
+\dX stts_?
+\dX *stts_hoge
+\dX+
+\dX+ stts_?
+\dX+ *stts_hoge
+\dX+ stts_s2.stts_yama
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
+
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
-- 
2.26.2


--------------B0278A9E952F6668D2DBEBA1
Content-Type: text/x-patch; charset=UTF-8;
 name="0002-fixup-rename-defined-to-requested-20210109.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="0002-fixup-rename-defined-to-requested-20210109.patch"



^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* [PATCH 1/2] psql \dX: list extended statistics objects
@ 2021-01-07 05:28 Tatsuro Yamada <[email protected]>
  0 siblings, 0 replies; 30+ messages in thread

From: Tatsuro Yamada @ 2021-01-07 05:28 UTC (permalink / raw)

The new command lists extended statistics objects, possibly with their
sizes. All past releases with extended statistics are supported.

Author: Tatsuro Yamada
Reviewed-by: Julien Rouhaud, Alvaro Herrera, Tomas Vondra
Discussion: https://postgr.es/m/c027a541-5856-75a5-0868-341301e1624b%40nttcom.co.jp_1
---
 doc/src/sgml/ref/psql-ref.sgml          |  14 +++
 src/bin/psql/command.c                  |   3 +
 src/bin/psql/describe.c                 | 150 ++++++++++++++++++++++++
 src/bin/psql/describe.h                 |   3 +
 src/bin/psql/help.c                     |   1 +
 src/bin/psql/tab-complete.c             |   4 +-
 src/test/regress/expected/stats_ext.out |  94 +++++++++++++++
 src/test/regress/sql/stats_ext.sql      |  31 +++++
 8 files changed, 299 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml
index 221a967bfe..d01acc92b8 100644
--- a/doc/src/sgml/ref/psql-ref.sgml
+++ b/doc/src/sgml/ref/psql-ref.sgml
@@ -1918,6 +1918,20 @@ testdb=&gt;
         </para>
         </listitem>
       </varlistentry>
+      
+      <varlistentry>
+        <term><literal>\dX [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
+        <listitem>
+        <para>
+        Lists extended statistics.
+        If <replaceable class="parameter">pattern</replaceable>
+        is specified, only those extended statistics whose names match the
+        pattern are listed.
+        If <literal>+</literal> is appended to the command name, each extended
+        statistics is listed with its size.
+        </para>
+        </listitem>
+      </varlistentry>
 
       <varlistentry>
         <term><literal>\dy[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index 303e7c3ad8..c5ebc1c3f4 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -928,6 +928,9 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd)
 				else
 					success = listExtensions(pattern);
 				break;
+			case 'X':			/* Extended Statistics */
+				success = listExtendedStats(pattern, show_verbose);
+				break;
 			case 'y':			/* Event Triggers */
 				success = listEventTriggers(pattern, show_verbose);
 				break;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index caf97563f4..46f54199fb 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -4392,6 +4392,156 @@ listEventTriggers(const char *pattern, bool verbose)
 	return true;
 }
 
+/*
+ * \dX
+ *
+ * Describes extended statistics.
+ */
+bool
+listExtendedStats(const char *pattern, bool verbose)
+{
+	PQExpBufferData buf;
+	PGresult   *res;
+	printQueryOpt myopt = pset.popt;
+
+	if (pset.sversion < 100000)
+	{
+		char		sverbuf[32];
+
+		pg_log_error("The server (version %s) does not support extended statistics.",
+					 formatPGVersionNumber(pset.sversion, false,
+										   sverbuf, sizeof(sverbuf)));
+		return true;
+	}
+
+	initPQExpBuffer(&buf);
+	printfPQExpBuffer(&buf,
+					  "SELECT \n"
+					  "es.stxnamespace::pg_catalog.regnamespace::text AS \"%s\", \n"
+					  "es.stxname AS \"%s\", \n"
+					  "pg_catalog.format('%%s FROM %%s', \n"
+					  "  (SELECT pg_catalog.string_agg(pg_catalog.quote_ident(a.attname),', ') \n"
+					  "   FROM pg_catalog.unnest(es.stxkeys) s(attnum) \n"
+					  "   JOIN pg_catalog.pg_attribute a \n"
+					  "   ON (es.stxrelid = a.attrelid \n"
+					  "   AND a.attnum = s.attnum \n"
+					  "   AND NOT a.attisdropped)), \n"
+					  "es.stxrelid::regclass) AS \"%s\"",
+					  gettext_noop("Schema"),
+					  gettext_noop("Name"),
+					  gettext_noop("Definition"));
+
+	/*
+	 * Since 12 there are two catalogs - one for the definition, one for the
+	 * data built by ANALYZE. Older releases use a single catalog. Also, 12
+	 * adds the MCV statistics kind.
+	 */
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN es.stxdependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"));
+	}
+	else
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxddependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxdmcv IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'm' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"),
+						  gettext_noop("MCV"));
+	}
+
+	/* In verbose mode, print sizes of the extended statistics objects. */
+	if (verbose)
+	{
+		if (pset.sversion < 120000)
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN es.stxdependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxdependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"));
+		}
+		else
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxddependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxddependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxdmcv IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdmcv)::bigint) \n"
+							  "     WHEN 'm' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"),
+							  gettext_noop("MCV_size"));
+		}
+	}
+
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+	else
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "LEFT JOIN pg_catalog.pg_statistic_ext_data esd \n"
+							 "ON es.oid = esd.stxoid \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+
+	processSQLNamePattern(pset.db, &buf, pattern, false,
+						  false, NULL,
+						  "stxname", NULL,
+						  NULL);
+
+	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
+
+	res = PSQLexec(buf.data);
+	termPQExpBuffer(&buf);
+	if (!res)
+		return false;
+
+	myopt.nullPrint = NULL;
+	myopt.title = _("List of extended statistics");
+	myopt.translate_header = true;
+
+	printQuery(res, &myopt, pset.queryFout, false, pset.logfile);
+
+	PQclear(res);
+	return true;
+}
+
 /*
  * \dC
  *
diff --git a/src/bin/psql/describe.h b/src/bin/psql/describe.h
index 6044e3a082..867e57d851 100644
--- a/src/bin/psql/describe.h
+++ b/src/bin/psql/describe.h
@@ -102,6 +102,9 @@ extern bool listExtensions(const char *pattern);
 /* \dx+ */
 extern bool listExtensionContents(const char *pattern);
 
+/* \dX */
+extern bool listExtendedStats(const char *pattern, bool verbose);
+
 /* \dy */
 extern bool listEventTriggers(const char *pattern, bool verbose);
 
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 9ec1c4e810..e42bc8c54e 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -267,6 +267,7 @@ slashUsage(unsigned short int pager)
 	fprintf(output, _("  \\du[S+] [PATTERN]      list roles\n"));
 	fprintf(output, _("  \\dv[S+] [PATTERN]      list views\n"));
 	fprintf(output, _("  \\dx[+]  [PATTERN]      list extensions\n"));
+	fprintf(output, _("  \\dX[+]  [PATTERN]      list extended statistics\n"));
 	fprintf(output, _("  \\dy     [PATTERN]      list event triggers\n"));
 	fprintf(output, _("  \\l[+]   [PATTERN]      list databases\n"));
 	fprintf(output, _("  \\sf[+]  FUNCNAME       show a function's definition\n"));
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9dcab0d2fa..611f1efb15 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1500,7 +1500,7 @@ psql_completion(const char *text, int start, int end)
 		"\\dF", "\\dFd", "\\dFp", "\\dFt", "\\dg", "\\di", "\\dl", "\\dL",
 		"\\dm", "\\dn", "\\do", "\\dO", "\\dp", "\\dP", "\\dPi", "\\dPt",
 		"\\drds", "\\dRs", "\\dRp", "\\ds", "\\dS",
-		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dy",
+		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dX", "\\dy",
 		"\\e", "\\echo", "\\ef", "\\elif", "\\else", "\\encoding",
 		"\\endif", "\\errverbose", "\\ev",
 		"\\f",
@@ -3910,6 +3910,8 @@ psql_completion(const char *text, int start, int end)
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_views, NULL);
 	else if (TailMatchesCS("\\dx*"))
 		COMPLETE_WITH_QUERY(Query_for_list_of_extensions);
+	else if (TailMatchesCS("\\dX*"))
+		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_statistics, NULL);
 	else if (TailMatchesCS("\\dm*"))
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_matviews, NULL);
 	else if (TailMatchesCS("\\dE*"))
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 7bfeaf85f0..8c8a0afcf6 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -1725,6 +1725,100 @@ INSERT INTO tststats.priv_test_tbl
 CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
   FROM tststats.priv_test_tbl;
 ANALYZE tststats.priv_test_tbl;
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+\dX
+                                          List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   
+----------+------------------------+--------------------------------------+-----------+--------------+---------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built
+(12 rows)
+
+\dX stts_?
+                       List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   
+--------+--------+-------------------+-----------+--------------+---------
+ public | stts_1 | a, b FROM stts_t1 | built     |              | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined
+(4 rows)
+
+\dX *stts_hoge
+                               List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   
+--------+-----------+-------------------------------+-----------+--------------+---------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined
+(1 row)
+
+\dX+
+                                                                   List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+----------+------------------------+--------------------------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        |         |                | 106 bytes         | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built   |                |                   | 24 kB
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built   |                |                   | 386 bytes
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built   |                |                   | 294 bytes
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              |         | 13 bytes       |                   | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined |                | 0 bytes           | 0 bytes
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built   |                |                   | 686 bytes
+(12 rows)
+
+\dX+ stts_?
+                                                List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+--------+--------+-------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public | stts_1 | a, b FROM stts_t1 | built     |              |         | 13 bytes       |                   | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(4 rows)
+
+\dX+ *stts_hoge
+                                                       List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+--------+-----------+-------------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(1 row)
+
+\dX+ stts_s2.stts_yama
+                                                    List of extended statistics
+ Schema  |   Name    |       Definition        | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+---------+-----------+-------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ stts_s2 | stts_yama | col1, col3 FROM stts_t3 |           | defined      | defined |                | 0 bytes           | 0 bytes
+(1 row)
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 7912e733ae..db6e3e1ba3 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -912,6 +912,37 @@ CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
 
 ANALYZE tststats.priv_test_tbl;
 
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+
+\dX
+\dX stts_?
+\dX *stts_hoge
+\dX+
+\dX+ stts_?
+\dX+ *stts_hoge
+\dX+ stts_s2.stts_yama
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
+
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
-- 
2.26.2


--------------B0278A9E952F6668D2DBEBA1
Content-Type: text/x-patch; charset=UTF-8;
 name="0002-fixup-rename-defined-to-requested-20210109.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="0002-fixup-rename-defined-to-requested-20210109.patch"



^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* [PATCH 1/2] psql \dX: list extended statistics objects
@ 2021-01-07 05:28 Tatsuro Yamada <[email protected]>
  0 siblings, 0 replies; 30+ messages in thread

From: Tatsuro Yamada @ 2021-01-07 05:28 UTC (permalink / raw)

The new command lists extended statistics objects, possibly with their
sizes. All past releases with extended statistics are supported.

Author: Tatsuro Yamada
Reviewed-by: Julien Rouhaud, Alvaro Herrera, Tomas Vondra
Discussion: https://postgr.es/m/c027a541-5856-75a5-0868-341301e1624b%40nttcom.co.jp_1
---
 doc/src/sgml/ref/psql-ref.sgml          |  14 +++
 src/bin/psql/command.c                  |   3 +
 src/bin/psql/describe.c                 | 150 ++++++++++++++++++++++++
 src/bin/psql/describe.h                 |   3 +
 src/bin/psql/help.c                     |   1 +
 src/bin/psql/tab-complete.c             |   4 +-
 src/test/regress/expected/stats_ext.out |  94 +++++++++++++++
 src/test/regress/sql/stats_ext.sql      |  31 +++++
 8 files changed, 299 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml
index 221a967bfe..d01acc92b8 100644
--- a/doc/src/sgml/ref/psql-ref.sgml
+++ b/doc/src/sgml/ref/psql-ref.sgml
@@ -1918,6 +1918,20 @@ testdb=&gt;
         </para>
         </listitem>
       </varlistentry>
+      
+      <varlistentry>
+        <term><literal>\dX [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
+        <listitem>
+        <para>
+        Lists extended statistics.
+        If <replaceable class="parameter">pattern</replaceable>
+        is specified, only those extended statistics whose names match the
+        pattern are listed.
+        If <literal>+</literal> is appended to the command name, each extended
+        statistics is listed with its size.
+        </para>
+        </listitem>
+      </varlistentry>
 
       <varlistentry>
         <term><literal>\dy[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index 303e7c3ad8..c5ebc1c3f4 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -928,6 +928,9 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd)
 				else
 					success = listExtensions(pattern);
 				break;
+			case 'X':			/* Extended Statistics */
+				success = listExtendedStats(pattern, show_verbose);
+				break;
 			case 'y':			/* Event Triggers */
 				success = listEventTriggers(pattern, show_verbose);
 				break;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index caf97563f4..46f54199fb 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -4392,6 +4392,156 @@ listEventTriggers(const char *pattern, bool verbose)
 	return true;
 }
 
+/*
+ * \dX
+ *
+ * Describes extended statistics.
+ */
+bool
+listExtendedStats(const char *pattern, bool verbose)
+{
+	PQExpBufferData buf;
+	PGresult   *res;
+	printQueryOpt myopt = pset.popt;
+
+	if (pset.sversion < 100000)
+	{
+		char		sverbuf[32];
+
+		pg_log_error("The server (version %s) does not support extended statistics.",
+					 formatPGVersionNumber(pset.sversion, false,
+										   sverbuf, sizeof(sverbuf)));
+		return true;
+	}
+
+	initPQExpBuffer(&buf);
+	printfPQExpBuffer(&buf,
+					  "SELECT \n"
+					  "es.stxnamespace::pg_catalog.regnamespace::text AS \"%s\", \n"
+					  "es.stxname AS \"%s\", \n"
+					  "pg_catalog.format('%%s FROM %%s', \n"
+					  "  (SELECT pg_catalog.string_agg(pg_catalog.quote_ident(a.attname),', ') \n"
+					  "   FROM pg_catalog.unnest(es.stxkeys) s(attnum) \n"
+					  "   JOIN pg_catalog.pg_attribute a \n"
+					  "   ON (es.stxrelid = a.attrelid \n"
+					  "   AND a.attnum = s.attnum \n"
+					  "   AND NOT a.attisdropped)), \n"
+					  "es.stxrelid::regclass) AS \"%s\"",
+					  gettext_noop("Schema"),
+					  gettext_noop("Name"),
+					  gettext_noop("Definition"));
+
+	/*
+	 * Since 12 there are two catalogs - one for the definition, one for the
+	 * data built by ANALYZE. Older releases use a single catalog. Also, 12
+	 * adds the MCV statistics kind.
+	 */
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN es.stxdependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"));
+	}
+	else
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxddependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxdmcv IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'm' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"),
+						  gettext_noop("MCV"));
+	}
+
+	/* In verbose mode, print sizes of the extended statistics objects. */
+	if (verbose)
+	{
+		if (pset.sversion < 120000)
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN es.stxdependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxdependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"));
+		}
+		else
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxddependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxddependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxdmcv IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdmcv)::bigint) \n"
+							  "     WHEN 'm' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"),
+							  gettext_noop("MCV_size"));
+		}
+	}
+
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+	else
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "LEFT JOIN pg_catalog.pg_statistic_ext_data esd \n"
+							 "ON es.oid = esd.stxoid \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+
+	processSQLNamePattern(pset.db, &buf, pattern, false,
+						  false, NULL,
+						  "stxname", NULL,
+						  NULL);
+
+	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
+
+	res = PSQLexec(buf.data);
+	termPQExpBuffer(&buf);
+	if (!res)
+		return false;
+
+	myopt.nullPrint = NULL;
+	myopt.title = _("List of extended statistics");
+	myopt.translate_header = true;
+
+	printQuery(res, &myopt, pset.queryFout, false, pset.logfile);
+
+	PQclear(res);
+	return true;
+}
+
 /*
  * \dC
  *
diff --git a/src/bin/psql/describe.h b/src/bin/psql/describe.h
index 6044e3a082..867e57d851 100644
--- a/src/bin/psql/describe.h
+++ b/src/bin/psql/describe.h
@@ -102,6 +102,9 @@ extern bool listExtensions(const char *pattern);
 /* \dx+ */
 extern bool listExtensionContents(const char *pattern);
 
+/* \dX */
+extern bool listExtendedStats(const char *pattern, bool verbose);
+
 /* \dy */
 extern bool listEventTriggers(const char *pattern, bool verbose);
 
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 9ec1c4e810..e42bc8c54e 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -267,6 +267,7 @@ slashUsage(unsigned short int pager)
 	fprintf(output, _("  \\du[S+] [PATTERN]      list roles\n"));
 	fprintf(output, _("  \\dv[S+] [PATTERN]      list views\n"));
 	fprintf(output, _("  \\dx[+]  [PATTERN]      list extensions\n"));
+	fprintf(output, _("  \\dX[+]  [PATTERN]      list extended statistics\n"));
 	fprintf(output, _("  \\dy     [PATTERN]      list event triggers\n"));
 	fprintf(output, _("  \\l[+]   [PATTERN]      list databases\n"));
 	fprintf(output, _("  \\sf[+]  FUNCNAME       show a function's definition\n"));
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9dcab0d2fa..611f1efb15 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1500,7 +1500,7 @@ psql_completion(const char *text, int start, int end)
 		"\\dF", "\\dFd", "\\dFp", "\\dFt", "\\dg", "\\di", "\\dl", "\\dL",
 		"\\dm", "\\dn", "\\do", "\\dO", "\\dp", "\\dP", "\\dPi", "\\dPt",
 		"\\drds", "\\dRs", "\\dRp", "\\ds", "\\dS",
-		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dy",
+		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dX", "\\dy",
 		"\\e", "\\echo", "\\ef", "\\elif", "\\else", "\\encoding",
 		"\\endif", "\\errverbose", "\\ev",
 		"\\f",
@@ -3910,6 +3910,8 @@ psql_completion(const char *text, int start, int end)
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_views, NULL);
 	else if (TailMatchesCS("\\dx*"))
 		COMPLETE_WITH_QUERY(Query_for_list_of_extensions);
+	else if (TailMatchesCS("\\dX*"))
+		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_statistics, NULL);
 	else if (TailMatchesCS("\\dm*"))
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_matviews, NULL);
 	else if (TailMatchesCS("\\dE*"))
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 7bfeaf85f0..8c8a0afcf6 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -1725,6 +1725,100 @@ INSERT INTO tststats.priv_test_tbl
 CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
   FROM tststats.priv_test_tbl;
 ANALYZE tststats.priv_test_tbl;
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+\dX
+                                          List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   
+----------+------------------------+--------------------------------------+-----------+--------------+---------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built
+(12 rows)
+
+\dX stts_?
+                       List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   
+--------+--------+-------------------+-----------+--------------+---------
+ public | stts_1 | a, b FROM stts_t1 | built     |              | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined
+(4 rows)
+
+\dX *stts_hoge
+                               List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   
+--------+-----------+-------------------------------+-----------+--------------+---------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined
+(1 row)
+
+\dX+
+                                                                   List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+----------+------------------------+--------------------------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        |         |                | 106 bytes         | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built   |                |                   | 24 kB
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built   |                |                   | 386 bytes
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built   |                |                   | 294 bytes
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              |         | 13 bytes       |                   | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined |                | 0 bytes           | 0 bytes
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built   |                |                   | 686 bytes
+(12 rows)
+
+\dX+ stts_?
+                                                List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+--------+--------+-------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public | stts_1 | a, b FROM stts_t1 | built     |              |         | 13 bytes       |                   | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(4 rows)
+
+\dX+ *stts_hoge
+                                                       List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+--------+-----------+-------------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(1 row)
+
+\dX+ stts_s2.stts_yama
+                                                    List of extended statistics
+ Schema  |   Name    |       Definition        | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+---------+-----------+-------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ stts_s2 | stts_yama | col1, col3 FROM stts_t3 |           | defined      | defined |                | 0 bytes           | 0 bytes
+(1 row)
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 7912e733ae..db6e3e1ba3 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -912,6 +912,37 @@ CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
 
 ANALYZE tststats.priv_test_tbl;
 
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+
+\dX
+\dX stts_?
+\dX *stts_hoge
+\dX+
+\dX+ stts_?
+\dX+ *stts_hoge
+\dX+ stts_s2.stts_yama
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
+
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
-- 
2.26.2


--------------B0278A9E952F6668D2DBEBA1
Content-Type: text/x-patch; charset=UTF-8;
 name="0002-fixup-rename-defined-to-requested-20210109.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="0002-fixup-rename-defined-to-requested-20210109.patch"



^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* [PATCH 1/2] psql \dX: list extended statistics objects
@ 2021-01-07 05:28 Tatsuro Yamada <[email protected]>
  0 siblings, 0 replies; 30+ messages in thread

From: Tatsuro Yamada @ 2021-01-07 05:28 UTC (permalink / raw)

The new command lists extended statistics objects, possibly with their
sizes. All past releases with extended statistics are supported.

Author: Tatsuro Yamada
Reviewed-by: Julien Rouhaud, Alvaro Herrera, Tomas Vondra
Discussion: https://postgr.es/m/c027a541-5856-75a5-0868-341301e1624b%40nttcom.co.jp_1
---
 doc/src/sgml/ref/psql-ref.sgml          |  14 +++
 src/bin/psql/command.c                  |   3 +
 src/bin/psql/describe.c                 | 150 ++++++++++++++++++++++++
 src/bin/psql/describe.h                 |   3 +
 src/bin/psql/help.c                     |   1 +
 src/bin/psql/tab-complete.c             |   4 +-
 src/test/regress/expected/stats_ext.out |  94 +++++++++++++++
 src/test/regress/sql/stats_ext.sql      |  31 +++++
 8 files changed, 299 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml
index 221a967bfe..d01acc92b8 100644
--- a/doc/src/sgml/ref/psql-ref.sgml
+++ b/doc/src/sgml/ref/psql-ref.sgml
@@ -1918,6 +1918,20 @@ testdb=&gt;
         </para>
         </listitem>
       </varlistentry>
+      
+      <varlistentry>
+        <term><literal>\dX [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
+        <listitem>
+        <para>
+        Lists extended statistics.
+        If <replaceable class="parameter">pattern</replaceable>
+        is specified, only those extended statistics whose names match the
+        pattern are listed.
+        If <literal>+</literal> is appended to the command name, each extended
+        statistics is listed with its size.
+        </para>
+        </listitem>
+      </varlistentry>
 
       <varlistentry>
         <term><literal>\dy[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index 303e7c3ad8..c5ebc1c3f4 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -928,6 +928,9 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd)
 				else
 					success = listExtensions(pattern);
 				break;
+			case 'X':			/* Extended Statistics */
+				success = listExtendedStats(pattern, show_verbose);
+				break;
 			case 'y':			/* Event Triggers */
 				success = listEventTriggers(pattern, show_verbose);
 				break;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index caf97563f4..46f54199fb 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -4392,6 +4392,156 @@ listEventTriggers(const char *pattern, bool verbose)
 	return true;
 }
 
+/*
+ * \dX
+ *
+ * Describes extended statistics.
+ */
+bool
+listExtendedStats(const char *pattern, bool verbose)
+{
+	PQExpBufferData buf;
+	PGresult   *res;
+	printQueryOpt myopt = pset.popt;
+
+	if (pset.sversion < 100000)
+	{
+		char		sverbuf[32];
+
+		pg_log_error("The server (version %s) does not support extended statistics.",
+					 formatPGVersionNumber(pset.sversion, false,
+										   sverbuf, sizeof(sverbuf)));
+		return true;
+	}
+
+	initPQExpBuffer(&buf);
+	printfPQExpBuffer(&buf,
+					  "SELECT \n"
+					  "es.stxnamespace::pg_catalog.regnamespace::text AS \"%s\", \n"
+					  "es.stxname AS \"%s\", \n"
+					  "pg_catalog.format('%%s FROM %%s', \n"
+					  "  (SELECT pg_catalog.string_agg(pg_catalog.quote_ident(a.attname),', ') \n"
+					  "   FROM pg_catalog.unnest(es.stxkeys) s(attnum) \n"
+					  "   JOIN pg_catalog.pg_attribute a \n"
+					  "   ON (es.stxrelid = a.attrelid \n"
+					  "   AND a.attnum = s.attnum \n"
+					  "   AND NOT a.attisdropped)), \n"
+					  "es.stxrelid::regclass) AS \"%s\"",
+					  gettext_noop("Schema"),
+					  gettext_noop("Name"),
+					  gettext_noop("Definition"));
+
+	/*
+	 * Since 12 there are two catalogs - one for the definition, one for the
+	 * data built by ANALYZE. Older releases use a single catalog. Also, 12
+	 * adds the MCV statistics kind.
+	 */
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN es.stxdependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"));
+	}
+	else
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxddependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxdmcv IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'm' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"),
+						  gettext_noop("MCV"));
+	}
+
+	/* In verbose mode, print sizes of the extended statistics objects. */
+	if (verbose)
+	{
+		if (pset.sversion < 120000)
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN es.stxdependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxdependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"));
+		}
+		else
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxddependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxddependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxdmcv IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdmcv)::bigint) \n"
+							  "     WHEN 'm' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"),
+							  gettext_noop("MCV_size"));
+		}
+	}
+
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+	else
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "LEFT JOIN pg_catalog.pg_statistic_ext_data esd \n"
+							 "ON es.oid = esd.stxoid \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+
+	processSQLNamePattern(pset.db, &buf, pattern, false,
+						  false, NULL,
+						  "stxname", NULL,
+						  NULL);
+
+	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
+
+	res = PSQLexec(buf.data);
+	termPQExpBuffer(&buf);
+	if (!res)
+		return false;
+
+	myopt.nullPrint = NULL;
+	myopt.title = _("List of extended statistics");
+	myopt.translate_header = true;
+
+	printQuery(res, &myopt, pset.queryFout, false, pset.logfile);
+
+	PQclear(res);
+	return true;
+}
+
 /*
  * \dC
  *
diff --git a/src/bin/psql/describe.h b/src/bin/psql/describe.h
index 6044e3a082..867e57d851 100644
--- a/src/bin/psql/describe.h
+++ b/src/bin/psql/describe.h
@@ -102,6 +102,9 @@ extern bool listExtensions(const char *pattern);
 /* \dx+ */
 extern bool listExtensionContents(const char *pattern);
 
+/* \dX */
+extern bool listExtendedStats(const char *pattern, bool verbose);
+
 /* \dy */
 extern bool listEventTriggers(const char *pattern, bool verbose);
 
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 9ec1c4e810..e42bc8c54e 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -267,6 +267,7 @@ slashUsage(unsigned short int pager)
 	fprintf(output, _("  \\du[S+] [PATTERN]      list roles\n"));
 	fprintf(output, _("  \\dv[S+] [PATTERN]      list views\n"));
 	fprintf(output, _("  \\dx[+]  [PATTERN]      list extensions\n"));
+	fprintf(output, _("  \\dX[+]  [PATTERN]      list extended statistics\n"));
 	fprintf(output, _("  \\dy     [PATTERN]      list event triggers\n"));
 	fprintf(output, _("  \\l[+]   [PATTERN]      list databases\n"));
 	fprintf(output, _("  \\sf[+]  FUNCNAME       show a function's definition\n"));
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9dcab0d2fa..611f1efb15 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1500,7 +1500,7 @@ psql_completion(const char *text, int start, int end)
 		"\\dF", "\\dFd", "\\dFp", "\\dFt", "\\dg", "\\di", "\\dl", "\\dL",
 		"\\dm", "\\dn", "\\do", "\\dO", "\\dp", "\\dP", "\\dPi", "\\dPt",
 		"\\drds", "\\dRs", "\\dRp", "\\ds", "\\dS",
-		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dy",
+		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dX", "\\dy",
 		"\\e", "\\echo", "\\ef", "\\elif", "\\else", "\\encoding",
 		"\\endif", "\\errverbose", "\\ev",
 		"\\f",
@@ -3910,6 +3910,8 @@ psql_completion(const char *text, int start, int end)
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_views, NULL);
 	else if (TailMatchesCS("\\dx*"))
 		COMPLETE_WITH_QUERY(Query_for_list_of_extensions);
+	else if (TailMatchesCS("\\dX*"))
+		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_statistics, NULL);
 	else if (TailMatchesCS("\\dm*"))
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_matviews, NULL);
 	else if (TailMatchesCS("\\dE*"))
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 7bfeaf85f0..8c8a0afcf6 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -1725,6 +1725,100 @@ INSERT INTO tststats.priv_test_tbl
 CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
   FROM tststats.priv_test_tbl;
 ANALYZE tststats.priv_test_tbl;
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+\dX
+                                          List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   
+----------+------------------------+--------------------------------------+-----------+--------------+---------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built
+(12 rows)
+
+\dX stts_?
+                       List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   
+--------+--------+-------------------+-----------+--------------+---------
+ public | stts_1 | a, b FROM stts_t1 | built     |              | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined
+(4 rows)
+
+\dX *stts_hoge
+                               List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   
+--------+-----------+-------------------------------+-----------+--------------+---------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined
+(1 row)
+
+\dX+
+                                                                   List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+----------+------------------------+--------------------------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        |         |                | 106 bytes         | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built   |                |                   | 24 kB
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built   |                |                   | 386 bytes
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built   |                |                   | 294 bytes
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              |         | 13 bytes       |                   | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined |                | 0 bytes           | 0 bytes
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built   |                |                   | 686 bytes
+(12 rows)
+
+\dX+ stts_?
+                                                List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+--------+--------+-------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public | stts_1 | a, b FROM stts_t1 | built     |              |         | 13 bytes       |                   | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(4 rows)
+
+\dX+ *stts_hoge
+                                                       List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+--------+-----------+-------------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(1 row)
+
+\dX+ stts_s2.stts_yama
+                                                    List of extended statistics
+ Schema  |   Name    |       Definition        | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+---------+-----------+-------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ stts_s2 | stts_yama | col1, col3 FROM stts_t3 |           | defined      | defined |                | 0 bytes           | 0 bytes
+(1 row)
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 7912e733ae..db6e3e1ba3 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -912,6 +912,37 @@ CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
 
 ANALYZE tststats.priv_test_tbl;
 
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+
+\dX
+\dX stts_?
+\dX *stts_hoge
+\dX+
+\dX+ stts_?
+\dX+ *stts_hoge
+\dX+ stts_s2.stts_yama
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
+
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
-- 
2.26.2


--------------B0278A9E952F6668D2DBEBA1
Content-Type: text/x-patch; charset=UTF-8;
 name="0002-fixup-rename-defined-to-requested-20210109.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="0002-fixup-rename-defined-to-requested-20210109.patch"



^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* [PATCH 1/2] psql \dX: list extended statistics objects
@ 2021-01-07 05:28 Tatsuro Yamada <[email protected]>
  0 siblings, 0 replies; 30+ messages in thread

From: Tatsuro Yamada @ 2021-01-07 05:28 UTC (permalink / raw)

The new command lists extended statistics objects, possibly with their
sizes. All past releases with extended statistics are supported.

Author: Tatsuro Yamada
Reviewed-by: Julien Rouhaud, Alvaro Herrera, Tomas Vondra
Discussion: https://postgr.es/m/c027a541-5856-75a5-0868-341301e1624b%40nttcom.co.jp_1
---
 doc/src/sgml/ref/psql-ref.sgml          |  14 +++
 src/bin/psql/command.c                  |   3 +
 src/bin/psql/describe.c                 | 150 ++++++++++++++++++++++++
 src/bin/psql/describe.h                 |   3 +
 src/bin/psql/help.c                     |   1 +
 src/bin/psql/tab-complete.c             |   4 +-
 src/test/regress/expected/stats_ext.out |  94 +++++++++++++++
 src/test/regress/sql/stats_ext.sql      |  31 +++++
 8 files changed, 299 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml
index 221a967bfe..d01acc92b8 100644
--- a/doc/src/sgml/ref/psql-ref.sgml
+++ b/doc/src/sgml/ref/psql-ref.sgml
@@ -1918,6 +1918,20 @@ testdb=&gt;
         </para>
         </listitem>
       </varlistentry>
+      
+      <varlistentry>
+        <term><literal>\dX [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
+        <listitem>
+        <para>
+        Lists extended statistics.
+        If <replaceable class="parameter">pattern</replaceable>
+        is specified, only those extended statistics whose names match the
+        pattern are listed.
+        If <literal>+</literal> is appended to the command name, each extended
+        statistics is listed with its size.
+        </para>
+        </listitem>
+      </varlistentry>
 
       <varlistentry>
         <term><literal>\dy[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index 303e7c3ad8..c5ebc1c3f4 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -928,6 +928,9 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd)
 				else
 					success = listExtensions(pattern);
 				break;
+			case 'X':			/* Extended Statistics */
+				success = listExtendedStats(pattern, show_verbose);
+				break;
 			case 'y':			/* Event Triggers */
 				success = listEventTriggers(pattern, show_verbose);
 				break;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index caf97563f4..46f54199fb 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -4392,6 +4392,156 @@ listEventTriggers(const char *pattern, bool verbose)
 	return true;
 }
 
+/*
+ * \dX
+ *
+ * Describes extended statistics.
+ */
+bool
+listExtendedStats(const char *pattern, bool verbose)
+{
+	PQExpBufferData buf;
+	PGresult   *res;
+	printQueryOpt myopt = pset.popt;
+
+	if (pset.sversion < 100000)
+	{
+		char		sverbuf[32];
+
+		pg_log_error("The server (version %s) does not support extended statistics.",
+					 formatPGVersionNumber(pset.sversion, false,
+										   sverbuf, sizeof(sverbuf)));
+		return true;
+	}
+
+	initPQExpBuffer(&buf);
+	printfPQExpBuffer(&buf,
+					  "SELECT \n"
+					  "es.stxnamespace::pg_catalog.regnamespace::text AS \"%s\", \n"
+					  "es.stxname AS \"%s\", \n"
+					  "pg_catalog.format('%%s FROM %%s', \n"
+					  "  (SELECT pg_catalog.string_agg(pg_catalog.quote_ident(a.attname),', ') \n"
+					  "   FROM pg_catalog.unnest(es.stxkeys) s(attnum) \n"
+					  "   JOIN pg_catalog.pg_attribute a \n"
+					  "   ON (es.stxrelid = a.attrelid \n"
+					  "   AND a.attnum = s.attnum \n"
+					  "   AND NOT a.attisdropped)), \n"
+					  "es.stxrelid::regclass) AS \"%s\"",
+					  gettext_noop("Schema"),
+					  gettext_noop("Name"),
+					  gettext_noop("Definition"));
+
+	/*
+	 * Since 12 there are two catalogs - one for the definition, one for the
+	 * data built by ANALYZE. Older releases use a single catalog. Also, 12
+	 * adds the MCV statistics kind.
+	 */
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN es.stxdependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"));
+	}
+	else
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxddependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxdmcv IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'm' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"),
+						  gettext_noop("MCV"));
+	}
+
+	/* In verbose mode, print sizes of the extended statistics objects. */
+	if (verbose)
+	{
+		if (pset.sversion < 120000)
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN es.stxdependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxdependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"));
+		}
+		else
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxddependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxddependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxdmcv IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdmcv)::bigint) \n"
+							  "     WHEN 'm' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"),
+							  gettext_noop("MCV_size"));
+		}
+	}
+
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+	else
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "LEFT JOIN pg_catalog.pg_statistic_ext_data esd \n"
+							 "ON es.oid = esd.stxoid \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+
+	processSQLNamePattern(pset.db, &buf, pattern, false,
+						  false, NULL,
+						  "stxname", NULL,
+						  NULL);
+
+	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
+
+	res = PSQLexec(buf.data);
+	termPQExpBuffer(&buf);
+	if (!res)
+		return false;
+
+	myopt.nullPrint = NULL;
+	myopt.title = _("List of extended statistics");
+	myopt.translate_header = true;
+
+	printQuery(res, &myopt, pset.queryFout, false, pset.logfile);
+
+	PQclear(res);
+	return true;
+}
+
 /*
  * \dC
  *
diff --git a/src/bin/psql/describe.h b/src/bin/psql/describe.h
index 6044e3a082..867e57d851 100644
--- a/src/bin/psql/describe.h
+++ b/src/bin/psql/describe.h
@@ -102,6 +102,9 @@ extern bool listExtensions(const char *pattern);
 /* \dx+ */
 extern bool listExtensionContents(const char *pattern);
 
+/* \dX */
+extern bool listExtendedStats(const char *pattern, bool verbose);
+
 /* \dy */
 extern bool listEventTriggers(const char *pattern, bool verbose);
 
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 9ec1c4e810..e42bc8c54e 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -267,6 +267,7 @@ slashUsage(unsigned short int pager)
 	fprintf(output, _("  \\du[S+] [PATTERN]      list roles\n"));
 	fprintf(output, _("  \\dv[S+] [PATTERN]      list views\n"));
 	fprintf(output, _("  \\dx[+]  [PATTERN]      list extensions\n"));
+	fprintf(output, _("  \\dX[+]  [PATTERN]      list extended statistics\n"));
 	fprintf(output, _("  \\dy     [PATTERN]      list event triggers\n"));
 	fprintf(output, _("  \\l[+]   [PATTERN]      list databases\n"));
 	fprintf(output, _("  \\sf[+]  FUNCNAME       show a function's definition\n"));
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9dcab0d2fa..611f1efb15 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1500,7 +1500,7 @@ psql_completion(const char *text, int start, int end)
 		"\\dF", "\\dFd", "\\dFp", "\\dFt", "\\dg", "\\di", "\\dl", "\\dL",
 		"\\dm", "\\dn", "\\do", "\\dO", "\\dp", "\\dP", "\\dPi", "\\dPt",
 		"\\drds", "\\dRs", "\\dRp", "\\ds", "\\dS",
-		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dy",
+		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dX", "\\dy",
 		"\\e", "\\echo", "\\ef", "\\elif", "\\else", "\\encoding",
 		"\\endif", "\\errverbose", "\\ev",
 		"\\f",
@@ -3910,6 +3910,8 @@ psql_completion(const char *text, int start, int end)
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_views, NULL);
 	else if (TailMatchesCS("\\dx*"))
 		COMPLETE_WITH_QUERY(Query_for_list_of_extensions);
+	else if (TailMatchesCS("\\dX*"))
+		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_statistics, NULL);
 	else if (TailMatchesCS("\\dm*"))
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_matviews, NULL);
 	else if (TailMatchesCS("\\dE*"))
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 7bfeaf85f0..8c8a0afcf6 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -1725,6 +1725,100 @@ INSERT INTO tststats.priv_test_tbl
 CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
   FROM tststats.priv_test_tbl;
 ANALYZE tststats.priv_test_tbl;
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+\dX
+                                          List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   
+----------+------------------------+--------------------------------------+-----------+--------------+---------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built
+(12 rows)
+
+\dX stts_?
+                       List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   
+--------+--------+-------------------+-----------+--------------+---------
+ public | stts_1 | a, b FROM stts_t1 | built     |              | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined
+(4 rows)
+
+\dX *stts_hoge
+                               List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   
+--------+-----------+-------------------------------+-----------+--------------+---------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined
+(1 row)
+
+\dX+
+                                                                   List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+----------+------------------------+--------------------------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        |         |                | 106 bytes         | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built   |                |                   | 24 kB
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built   |                |                   | 386 bytes
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built   |                |                   | 294 bytes
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              |         | 13 bytes       |                   | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined |                | 0 bytes           | 0 bytes
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built   |                |                   | 686 bytes
+(12 rows)
+
+\dX+ stts_?
+                                                List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+--------+--------+-------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public | stts_1 | a, b FROM stts_t1 | built     |              |         | 13 bytes       |                   | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(4 rows)
+
+\dX+ *stts_hoge
+                                                       List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+--------+-----------+-------------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(1 row)
+
+\dX+ stts_s2.stts_yama
+                                                    List of extended statistics
+ Schema  |   Name    |       Definition        | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+---------+-----------+-------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ stts_s2 | stts_yama | col1, col3 FROM stts_t3 |           | defined      | defined |                | 0 bytes           | 0 bytes
+(1 row)
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 7912e733ae..db6e3e1ba3 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -912,6 +912,37 @@ CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
 
 ANALYZE tststats.priv_test_tbl;
 
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+
+\dX
+\dX stts_?
+\dX *stts_hoge
+\dX+
+\dX+ stts_?
+\dX+ *stts_hoge
+\dX+ stts_s2.stts_yama
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
+
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
-- 
2.26.2


--------------B0278A9E952F6668D2DBEBA1
Content-Type: text/x-patch; charset=UTF-8;
 name="0002-fixup-rename-defined-to-requested-20210109.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="0002-fixup-rename-defined-to-requested-20210109.patch"



^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* [PATCH 1/2] psql \dX: list extended statistics objects
@ 2021-01-07 05:28 Tatsuro Yamada <[email protected]>
  0 siblings, 0 replies; 30+ messages in thread

From: Tatsuro Yamada @ 2021-01-07 05:28 UTC (permalink / raw)

The new command lists extended statistics objects, possibly with their
sizes. All past releases with extended statistics are supported.

Author: Tatsuro Yamada
Reviewed-by: Julien Rouhaud, Alvaro Herrera, Tomas Vondra
Discussion: https://postgr.es/m/c027a541-5856-75a5-0868-341301e1624b%40nttcom.co.jp_1
---
 doc/src/sgml/ref/psql-ref.sgml          |  14 +++
 src/bin/psql/command.c                  |   3 +
 src/bin/psql/describe.c                 | 150 ++++++++++++++++++++++++
 src/bin/psql/describe.h                 |   3 +
 src/bin/psql/help.c                     |   1 +
 src/bin/psql/tab-complete.c             |   4 +-
 src/test/regress/expected/stats_ext.out |  94 +++++++++++++++
 src/test/regress/sql/stats_ext.sql      |  31 +++++
 8 files changed, 299 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml
index 221a967bfe..d01acc92b8 100644
--- a/doc/src/sgml/ref/psql-ref.sgml
+++ b/doc/src/sgml/ref/psql-ref.sgml
@@ -1918,6 +1918,20 @@ testdb=&gt;
         </para>
         </listitem>
       </varlistentry>
+      
+      <varlistentry>
+        <term><literal>\dX [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
+        <listitem>
+        <para>
+        Lists extended statistics.
+        If <replaceable class="parameter">pattern</replaceable>
+        is specified, only those extended statistics whose names match the
+        pattern are listed.
+        If <literal>+</literal> is appended to the command name, each extended
+        statistics is listed with its size.
+        </para>
+        </listitem>
+      </varlistentry>
 
       <varlistentry>
         <term><literal>\dy[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index 303e7c3ad8..c5ebc1c3f4 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -928,6 +928,9 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd)
 				else
 					success = listExtensions(pattern);
 				break;
+			case 'X':			/* Extended Statistics */
+				success = listExtendedStats(pattern, show_verbose);
+				break;
 			case 'y':			/* Event Triggers */
 				success = listEventTriggers(pattern, show_verbose);
 				break;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index caf97563f4..46f54199fb 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -4392,6 +4392,156 @@ listEventTriggers(const char *pattern, bool verbose)
 	return true;
 }
 
+/*
+ * \dX
+ *
+ * Describes extended statistics.
+ */
+bool
+listExtendedStats(const char *pattern, bool verbose)
+{
+	PQExpBufferData buf;
+	PGresult   *res;
+	printQueryOpt myopt = pset.popt;
+
+	if (pset.sversion < 100000)
+	{
+		char		sverbuf[32];
+
+		pg_log_error("The server (version %s) does not support extended statistics.",
+					 formatPGVersionNumber(pset.sversion, false,
+										   sverbuf, sizeof(sverbuf)));
+		return true;
+	}
+
+	initPQExpBuffer(&buf);
+	printfPQExpBuffer(&buf,
+					  "SELECT \n"
+					  "es.stxnamespace::pg_catalog.regnamespace::text AS \"%s\", \n"
+					  "es.stxname AS \"%s\", \n"
+					  "pg_catalog.format('%%s FROM %%s', \n"
+					  "  (SELECT pg_catalog.string_agg(pg_catalog.quote_ident(a.attname),', ') \n"
+					  "   FROM pg_catalog.unnest(es.stxkeys) s(attnum) \n"
+					  "   JOIN pg_catalog.pg_attribute a \n"
+					  "   ON (es.stxrelid = a.attrelid \n"
+					  "   AND a.attnum = s.attnum \n"
+					  "   AND NOT a.attisdropped)), \n"
+					  "es.stxrelid::regclass) AS \"%s\"",
+					  gettext_noop("Schema"),
+					  gettext_noop("Name"),
+					  gettext_noop("Definition"));
+
+	/*
+	 * Since 12 there are two catalogs - one for the definition, one for the
+	 * data built by ANALYZE. Older releases use a single catalog. Also, 12
+	 * adds the MCV statistics kind.
+	 */
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN es.stxdependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"));
+	}
+	else
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxddependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxdmcv IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'm' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"),
+						  gettext_noop("MCV"));
+	}
+
+	/* In verbose mode, print sizes of the extended statistics objects. */
+	if (verbose)
+	{
+		if (pset.sversion < 120000)
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN es.stxdependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxdependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"));
+		}
+		else
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxddependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxddependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxdmcv IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdmcv)::bigint) \n"
+							  "     WHEN 'm' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"),
+							  gettext_noop("MCV_size"));
+		}
+	}
+
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+	else
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "LEFT JOIN pg_catalog.pg_statistic_ext_data esd \n"
+							 "ON es.oid = esd.stxoid \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+
+	processSQLNamePattern(pset.db, &buf, pattern, false,
+						  false, NULL,
+						  "stxname", NULL,
+						  NULL);
+
+	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
+
+	res = PSQLexec(buf.data);
+	termPQExpBuffer(&buf);
+	if (!res)
+		return false;
+
+	myopt.nullPrint = NULL;
+	myopt.title = _("List of extended statistics");
+	myopt.translate_header = true;
+
+	printQuery(res, &myopt, pset.queryFout, false, pset.logfile);
+
+	PQclear(res);
+	return true;
+}
+
 /*
  * \dC
  *
diff --git a/src/bin/psql/describe.h b/src/bin/psql/describe.h
index 6044e3a082..867e57d851 100644
--- a/src/bin/psql/describe.h
+++ b/src/bin/psql/describe.h
@@ -102,6 +102,9 @@ extern bool listExtensions(const char *pattern);
 /* \dx+ */
 extern bool listExtensionContents(const char *pattern);
 
+/* \dX */
+extern bool listExtendedStats(const char *pattern, bool verbose);
+
 /* \dy */
 extern bool listEventTriggers(const char *pattern, bool verbose);
 
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 9ec1c4e810..e42bc8c54e 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -267,6 +267,7 @@ slashUsage(unsigned short int pager)
 	fprintf(output, _("  \\du[S+] [PATTERN]      list roles\n"));
 	fprintf(output, _("  \\dv[S+] [PATTERN]      list views\n"));
 	fprintf(output, _("  \\dx[+]  [PATTERN]      list extensions\n"));
+	fprintf(output, _("  \\dX[+]  [PATTERN]      list extended statistics\n"));
 	fprintf(output, _("  \\dy     [PATTERN]      list event triggers\n"));
 	fprintf(output, _("  \\l[+]   [PATTERN]      list databases\n"));
 	fprintf(output, _("  \\sf[+]  FUNCNAME       show a function's definition\n"));
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9dcab0d2fa..611f1efb15 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1500,7 +1500,7 @@ psql_completion(const char *text, int start, int end)
 		"\\dF", "\\dFd", "\\dFp", "\\dFt", "\\dg", "\\di", "\\dl", "\\dL",
 		"\\dm", "\\dn", "\\do", "\\dO", "\\dp", "\\dP", "\\dPi", "\\dPt",
 		"\\drds", "\\dRs", "\\dRp", "\\ds", "\\dS",
-		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dy",
+		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dX", "\\dy",
 		"\\e", "\\echo", "\\ef", "\\elif", "\\else", "\\encoding",
 		"\\endif", "\\errverbose", "\\ev",
 		"\\f",
@@ -3910,6 +3910,8 @@ psql_completion(const char *text, int start, int end)
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_views, NULL);
 	else if (TailMatchesCS("\\dx*"))
 		COMPLETE_WITH_QUERY(Query_for_list_of_extensions);
+	else if (TailMatchesCS("\\dX*"))
+		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_statistics, NULL);
 	else if (TailMatchesCS("\\dm*"))
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_matviews, NULL);
 	else if (TailMatchesCS("\\dE*"))
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 7bfeaf85f0..8c8a0afcf6 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -1725,6 +1725,100 @@ INSERT INTO tststats.priv_test_tbl
 CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
   FROM tststats.priv_test_tbl;
 ANALYZE tststats.priv_test_tbl;
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+\dX
+                                          List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   
+----------+------------------------+--------------------------------------+-----------+--------------+---------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built
+(12 rows)
+
+\dX stts_?
+                       List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   
+--------+--------+-------------------+-----------+--------------+---------
+ public | stts_1 | a, b FROM stts_t1 | built     |              | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined
+(4 rows)
+
+\dX *stts_hoge
+                               List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   
+--------+-----------+-------------------------------+-----------+--------------+---------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined
+(1 row)
+
+\dX+
+                                                                   List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+----------+------------------------+--------------------------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        |         |                | 106 bytes         | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built   |                |                   | 24 kB
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built   |                |                   | 386 bytes
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built   |                |                   | 294 bytes
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              |         | 13 bytes       |                   | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined |                | 0 bytes           | 0 bytes
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built   |                |                   | 686 bytes
+(12 rows)
+
+\dX+ stts_?
+                                                List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+--------+--------+-------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public | stts_1 | a, b FROM stts_t1 | built     |              |         | 13 bytes       |                   | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(4 rows)
+
+\dX+ *stts_hoge
+                                                       List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+--------+-----------+-------------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(1 row)
+
+\dX+ stts_s2.stts_yama
+                                                    List of extended statistics
+ Schema  |   Name    |       Definition        | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+---------+-----------+-------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ stts_s2 | stts_yama | col1, col3 FROM stts_t3 |           | defined      | defined |                | 0 bytes           | 0 bytes
+(1 row)
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 7912e733ae..db6e3e1ba3 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -912,6 +912,37 @@ CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
 
 ANALYZE tststats.priv_test_tbl;
 
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+
+\dX
+\dX stts_?
+\dX *stts_hoge
+\dX+
+\dX+ stts_?
+\dX+ *stts_hoge
+\dX+ stts_s2.stts_yama
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
+
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
-- 
2.26.2


--------------B0278A9E952F6668D2DBEBA1
Content-Type: text/x-patch; charset=UTF-8;
 name="0002-fixup-rename-defined-to-requested-20210109.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="0002-fixup-rename-defined-to-requested-20210109.patch"



^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* [PATCH 1/2] psql \dX: list extended statistics objects
@ 2021-01-07 05:28 Tatsuro Yamada <[email protected]>
  0 siblings, 0 replies; 30+ messages in thread

From: Tatsuro Yamada @ 2021-01-07 05:28 UTC (permalink / raw)

The new command lists extended statistics objects, possibly with their
sizes. All past releases with extended statistics are supported.

Author: Tatsuro Yamada
Reviewed-by: Julien Rouhaud, Alvaro Herrera, Tomas Vondra
Discussion: https://postgr.es/m/c027a541-5856-75a5-0868-341301e1624b%40nttcom.co.jp_1
---
 doc/src/sgml/ref/psql-ref.sgml          |  14 +++
 src/bin/psql/command.c                  |   3 +
 src/bin/psql/describe.c                 | 150 ++++++++++++++++++++++++
 src/bin/psql/describe.h                 |   3 +
 src/bin/psql/help.c                     |   1 +
 src/bin/psql/tab-complete.c             |   4 +-
 src/test/regress/expected/stats_ext.out |  94 +++++++++++++++
 src/test/regress/sql/stats_ext.sql      |  31 +++++
 8 files changed, 299 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml
index 221a967bfe..d01acc92b8 100644
--- a/doc/src/sgml/ref/psql-ref.sgml
+++ b/doc/src/sgml/ref/psql-ref.sgml
@@ -1918,6 +1918,20 @@ testdb=&gt;
         </para>
         </listitem>
       </varlistentry>
+      
+      <varlistentry>
+        <term><literal>\dX [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
+        <listitem>
+        <para>
+        Lists extended statistics.
+        If <replaceable class="parameter">pattern</replaceable>
+        is specified, only those extended statistics whose names match the
+        pattern are listed.
+        If <literal>+</literal> is appended to the command name, each extended
+        statistics is listed with its size.
+        </para>
+        </listitem>
+      </varlistentry>
 
       <varlistentry>
         <term><literal>\dy[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index 303e7c3ad8..c5ebc1c3f4 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -928,6 +928,9 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd)
 				else
 					success = listExtensions(pattern);
 				break;
+			case 'X':			/* Extended Statistics */
+				success = listExtendedStats(pattern, show_verbose);
+				break;
 			case 'y':			/* Event Triggers */
 				success = listEventTriggers(pattern, show_verbose);
 				break;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index caf97563f4..46f54199fb 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -4392,6 +4392,156 @@ listEventTriggers(const char *pattern, bool verbose)
 	return true;
 }
 
+/*
+ * \dX
+ *
+ * Describes extended statistics.
+ */
+bool
+listExtendedStats(const char *pattern, bool verbose)
+{
+	PQExpBufferData buf;
+	PGresult   *res;
+	printQueryOpt myopt = pset.popt;
+
+	if (pset.sversion < 100000)
+	{
+		char		sverbuf[32];
+
+		pg_log_error("The server (version %s) does not support extended statistics.",
+					 formatPGVersionNumber(pset.sversion, false,
+										   sverbuf, sizeof(sverbuf)));
+		return true;
+	}
+
+	initPQExpBuffer(&buf);
+	printfPQExpBuffer(&buf,
+					  "SELECT \n"
+					  "es.stxnamespace::pg_catalog.regnamespace::text AS \"%s\", \n"
+					  "es.stxname AS \"%s\", \n"
+					  "pg_catalog.format('%%s FROM %%s', \n"
+					  "  (SELECT pg_catalog.string_agg(pg_catalog.quote_ident(a.attname),', ') \n"
+					  "   FROM pg_catalog.unnest(es.stxkeys) s(attnum) \n"
+					  "   JOIN pg_catalog.pg_attribute a \n"
+					  "   ON (es.stxrelid = a.attrelid \n"
+					  "   AND a.attnum = s.attnum \n"
+					  "   AND NOT a.attisdropped)), \n"
+					  "es.stxrelid::regclass) AS \"%s\"",
+					  gettext_noop("Schema"),
+					  gettext_noop("Name"),
+					  gettext_noop("Definition"));
+
+	/*
+	 * Since 12 there are two catalogs - one for the definition, one for the
+	 * data built by ANALYZE. Older releases use a single catalog. Also, 12
+	 * adds the MCV statistics kind.
+	 */
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN es.stxdependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"));
+	}
+	else
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxddependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxdmcv IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'm' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"),
+						  gettext_noop("MCV"));
+	}
+
+	/* In verbose mode, print sizes of the extended statistics objects. */
+	if (verbose)
+	{
+		if (pset.sversion < 120000)
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN es.stxdependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxdependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"));
+		}
+		else
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxddependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxddependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxdmcv IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdmcv)::bigint) \n"
+							  "     WHEN 'm' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"),
+							  gettext_noop("MCV_size"));
+		}
+	}
+
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+	else
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "LEFT JOIN pg_catalog.pg_statistic_ext_data esd \n"
+							 "ON es.oid = esd.stxoid \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+
+	processSQLNamePattern(pset.db, &buf, pattern, false,
+						  false, NULL,
+						  "stxname", NULL,
+						  NULL);
+
+	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
+
+	res = PSQLexec(buf.data);
+	termPQExpBuffer(&buf);
+	if (!res)
+		return false;
+
+	myopt.nullPrint = NULL;
+	myopt.title = _("List of extended statistics");
+	myopt.translate_header = true;
+
+	printQuery(res, &myopt, pset.queryFout, false, pset.logfile);
+
+	PQclear(res);
+	return true;
+}
+
 /*
  * \dC
  *
diff --git a/src/bin/psql/describe.h b/src/bin/psql/describe.h
index 6044e3a082..867e57d851 100644
--- a/src/bin/psql/describe.h
+++ b/src/bin/psql/describe.h
@@ -102,6 +102,9 @@ extern bool listExtensions(const char *pattern);
 /* \dx+ */
 extern bool listExtensionContents(const char *pattern);
 
+/* \dX */
+extern bool listExtendedStats(const char *pattern, bool verbose);
+
 /* \dy */
 extern bool listEventTriggers(const char *pattern, bool verbose);
 
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 9ec1c4e810..e42bc8c54e 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -267,6 +267,7 @@ slashUsage(unsigned short int pager)
 	fprintf(output, _("  \\du[S+] [PATTERN]      list roles\n"));
 	fprintf(output, _("  \\dv[S+] [PATTERN]      list views\n"));
 	fprintf(output, _("  \\dx[+]  [PATTERN]      list extensions\n"));
+	fprintf(output, _("  \\dX[+]  [PATTERN]      list extended statistics\n"));
 	fprintf(output, _("  \\dy     [PATTERN]      list event triggers\n"));
 	fprintf(output, _("  \\l[+]   [PATTERN]      list databases\n"));
 	fprintf(output, _("  \\sf[+]  FUNCNAME       show a function's definition\n"));
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9dcab0d2fa..611f1efb15 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1500,7 +1500,7 @@ psql_completion(const char *text, int start, int end)
 		"\\dF", "\\dFd", "\\dFp", "\\dFt", "\\dg", "\\di", "\\dl", "\\dL",
 		"\\dm", "\\dn", "\\do", "\\dO", "\\dp", "\\dP", "\\dPi", "\\dPt",
 		"\\drds", "\\dRs", "\\dRp", "\\ds", "\\dS",
-		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dy",
+		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dX", "\\dy",
 		"\\e", "\\echo", "\\ef", "\\elif", "\\else", "\\encoding",
 		"\\endif", "\\errverbose", "\\ev",
 		"\\f",
@@ -3910,6 +3910,8 @@ psql_completion(const char *text, int start, int end)
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_views, NULL);
 	else if (TailMatchesCS("\\dx*"))
 		COMPLETE_WITH_QUERY(Query_for_list_of_extensions);
+	else if (TailMatchesCS("\\dX*"))
+		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_statistics, NULL);
 	else if (TailMatchesCS("\\dm*"))
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_matviews, NULL);
 	else if (TailMatchesCS("\\dE*"))
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 7bfeaf85f0..8c8a0afcf6 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -1725,6 +1725,100 @@ INSERT INTO tststats.priv_test_tbl
 CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
   FROM tststats.priv_test_tbl;
 ANALYZE tststats.priv_test_tbl;
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+\dX
+                                          List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   
+----------+------------------------+--------------------------------------+-----------+--------------+---------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built
+(12 rows)
+
+\dX stts_?
+                       List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   
+--------+--------+-------------------+-----------+--------------+---------
+ public | stts_1 | a, b FROM stts_t1 | built     |              | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined
+(4 rows)
+
+\dX *stts_hoge
+                               List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   
+--------+-----------+-------------------------------+-----------+--------------+---------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined
+(1 row)
+
+\dX+
+                                                                   List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+----------+------------------------+--------------------------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        |         |                | 106 bytes         | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built   |                |                   | 24 kB
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built   |                |                   | 386 bytes
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built   |                |                   | 294 bytes
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              |         | 13 bytes       |                   | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined |                | 0 bytes           | 0 bytes
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built   |                |                   | 686 bytes
+(12 rows)
+
+\dX+ stts_?
+                                                List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+--------+--------+-------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public | stts_1 | a, b FROM stts_t1 | built     |              |         | 13 bytes       |                   | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(4 rows)
+
+\dX+ *stts_hoge
+                                                       List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+--------+-----------+-------------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(1 row)
+
+\dX+ stts_s2.stts_yama
+                                                    List of extended statistics
+ Schema  |   Name    |       Definition        | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+---------+-----------+-------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ stts_s2 | stts_yama | col1, col3 FROM stts_t3 |           | defined      | defined |                | 0 bytes           | 0 bytes
+(1 row)
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 7912e733ae..db6e3e1ba3 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -912,6 +912,37 @@ CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
 
 ANALYZE tststats.priv_test_tbl;
 
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+
+\dX
+\dX stts_?
+\dX *stts_hoge
+\dX+
+\dX+ stts_?
+\dX+ *stts_hoge
+\dX+ stts_s2.stts_yama
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
+
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
-- 
2.26.2


--------------B0278A9E952F6668D2DBEBA1
Content-Type: text/x-patch; charset=UTF-8;
 name="0002-fixup-rename-defined-to-requested-20210109.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="0002-fixup-rename-defined-to-requested-20210109.patch"



^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* [PATCH 1/2] psql \dX: list extended statistics objects
@ 2021-01-07 05:28 Tatsuro Yamada <[email protected]>
  0 siblings, 0 replies; 30+ messages in thread

From: Tatsuro Yamada @ 2021-01-07 05:28 UTC (permalink / raw)

The new command lists extended statistics objects, possibly with their
sizes. All past releases with extended statistics are supported.

Author: Tatsuro Yamada
Reviewed-by: Julien Rouhaud, Alvaro Herrera, Tomas Vondra
Discussion: https://postgr.es/m/c027a541-5856-75a5-0868-341301e1624b%40nttcom.co.jp_1
---
 doc/src/sgml/ref/psql-ref.sgml          |  14 +++
 src/bin/psql/command.c                  |   3 +
 src/bin/psql/describe.c                 | 150 ++++++++++++++++++++++++
 src/bin/psql/describe.h                 |   3 +
 src/bin/psql/help.c                     |   1 +
 src/bin/psql/tab-complete.c             |   4 +-
 src/test/regress/expected/stats_ext.out |  94 +++++++++++++++
 src/test/regress/sql/stats_ext.sql      |  31 +++++
 8 files changed, 299 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml
index 221a967bfe..d01acc92b8 100644
--- a/doc/src/sgml/ref/psql-ref.sgml
+++ b/doc/src/sgml/ref/psql-ref.sgml
@@ -1918,6 +1918,20 @@ testdb=&gt;
         </para>
         </listitem>
       </varlistentry>
+      
+      <varlistentry>
+        <term><literal>\dX [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
+        <listitem>
+        <para>
+        Lists extended statistics.
+        If <replaceable class="parameter">pattern</replaceable>
+        is specified, only those extended statistics whose names match the
+        pattern are listed.
+        If <literal>+</literal> is appended to the command name, each extended
+        statistics is listed with its size.
+        </para>
+        </listitem>
+      </varlistentry>
 
       <varlistentry>
         <term><literal>\dy[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index 303e7c3ad8..c5ebc1c3f4 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -928,6 +928,9 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd)
 				else
 					success = listExtensions(pattern);
 				break;
+			case 'X':			/* Extended Statistics */
+				success = listExtendedStats(pattern, show_verbose);
+				break;
 			case 'y':			/* Event Triggers */
 				success = listEventTriggers(pattern, show_verbose);
 				break;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index caf97563f4..46f54199fb 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -4392,6 +4392,156 @@ listEventTriggers(const char *pattern, bool verbose)
 	return true;
 }
 
+/*
+ * \dX
+ *
+ * Describes extended statistics.
+ */
+bool
+listExtendedStats(const char *pattern, bool verbose)
+{
+	PQExpBufferData buf;
+	PGresult   *res;
+	printQueryOpt myopt = pset.popt;
+
+	if (pset.sversion < 100000)
+	{
+		char		sverbuf[32];
+
+		pg_log_error("The server (version %s) does not support extended statistics.",
+					 formatPGVersionNumber(pset.sversion, false,
+										   sverbuf, sizeof(sverbuf)));
+		return true;
+	}
+
+	initPQExpBuffer(&buf);
+	printfPQExpBuffer(&buf,
+					  "SELECT \n"
+					  "es.stxnamespace::pg_catalog.regnamespace::text AS \"%s\", \n"
+					  "es.stxname AS \"%s\", \n"
+					  "pg_catalog.format('%%s FROM %%s', \n"
+					  "  (SELECT pg_catalog.string_agg(pg_catalog.quote_ident(a.attname),', ') \n"
+					  "   FROM pg_catalog.unnest(es.stxkeys) s(attnum) \n"
+					  "   JOIN pg_catalog.pg_attribute a \n"
+					  "   ON (es.stxrelid = a.attrelid \n"
+					  "   AND a.attnum = s.attnum \n"
+					  "   AND NOT a.attisdropped)), \n"
+					  "es.stxrelid::regclass) AS \"%s\"",
+					  gettext_noop("Schema"),
+					  gettext_noop("Name"),
+					  gettext_noop("Definition"));
+
+	/*
+	 * Since 12 there are two catalogs - one for the definition, one for the
+	 * data built by ANALYZE. Older releases use a single catalog. Also, 12
+	 * adds the MCV statistics kind.
+	 */
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN es.stxdependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"));
+	}
+	else
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxddependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxdmcv IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'm' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"),
+						  gettext_noop("MCV"));
+	}
+
+	/* In verbose mode, print sizes of the extended statistics objects. */
+	if (verbose)
+	{
+		if (pset.sversion < 120000)
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN es.stxdependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxdependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"));
+		}
+		else
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxddependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxddependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxdmcv IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdmcv)::bigint) \n"
+							  "     WHEN 'm' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"),
+							  gettext_noop("MCV_size"));
+		}
+	}
+
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+	else
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "LEFT JOIN pg_catalog.pg_statistic_ext_data esd \n"
+							 "ON es.oid = esd.stxoid \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+
+	processSQLNamePattern(pset.db, &buf, pattern, false,
+						  false, NULL,
+						  "stxname", NULL,
+						  NULL);
+
+	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
+
+	res = PSQLexec(buf.data);
+	termPQExpBuffer(&buf);
+	if (!res)
+		return false;
+
+	myopt.nullPrint = NULL;
+	myopt.title = _("List of extended statistics");
+	myopt.translate_header = true;
+
+	printQuery(res, &myopt, pset.queryFout, false, pset.logfile);
+
+	PQclear(res);
+	return true;
+}
+
 /*
  * \dC
  *
diff --git a/src/bin/psql/describe.h b/src/bin/psql/describe.h
index 6044e3a082..867e57d851 100644
--- a/src/bin/psql/describe.h
+++ b/src/bin/psql/describe.h
@@ -102,6 +102,9 @@ extern bool listExtensions(const char *pattern);
 /* \dx+ */
 extern bool listExtensionContents(const char *pattern);
 
+/* \dX */
+extern bool listExtendedStats(const char *pattern, bool verbose);
+
 /* \dy */
 extern bool listEventTriggers(const char *pattern, bool verbose);
 
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 9ec1c4e810..e42bc8c54e 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -267,6 +267,7 @@ slashUsage(unsigned short int pager)
 	fprintf(output, _("  \\du[S+] [PATTERN]      list roles\n"));
 	fprintf(output, _("  \\dv[S+] [PATTERN]      list views\n"));
 	fprintf(output, _("  \\dx[+]  [PATTERN]      list extensions\n"));
+	fprintf(output, _("  \\dX[+]  [PATTERN]      list extended statistics\n"));
 	fprintf(output, _("  \\dy     [PATTERN]      list event triggers\n"));
 	fprintf(output, _("  \\l[+]   [PATTERN]      list databases\n"));
 	fprintf(output, _("  \\sf[+]  FUNCNAME       show a function's definition\n"));
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9dcab0d2fa..611f1efb15 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1500,7 +1500,7 @@ psql_completion(const char *text, int start, int end)
 		"\\dF", "\\dFd", "\\dFp", "\\dFt", "\\dg", "\\di", "\\dl", "\\dL",
 		"\\dm", "\\dn", "\\do", "\\dO", "\\dp", "\\dP", "\\dPi", "\\dPt",
 		"\\drds", "\\dRs", "\\dRp", "\\ds", "\\dS",
-		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dy",
+		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dX", "\\dy",
 		"\\e", "\\echo", "\\ef", "\\elif", "\\else", "\\encoding",
 		"\\endif", "\\errverbose", "\\ev",
 		"\\f",
@@ -3910,6 +3910,8 @@ psql_completion(const char *text, int start, int end)
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_views, NULL);
 	else if (TailMatchesCS("\\dx*"))
 		COMPLETE_WITH_QUERY(Query_for_list_of_extensions);
+	else if (TailMatchesCS("\\dX*"))
+		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_statistics, NULL);
 	else if (TailMatchesCS("\\dm*"))
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_matviews, NULL);
 	else if (TailMatchesCS("\\dE*"))
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 7bfeaf85f0..8c8a0afcf6 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -1725,6 +1725,100 @@ INSERT INTO tststats.priv_test_tbl
 CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
   FROM tststats.priv_test_tbl;
 ANALYZE tststats.priv_test_tbl;
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+\dX
+                                          List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   
+----------+------------------------+--------------------------------------+-----------+--------------+---------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built
+(12 rows)
+
+\dX stts_?
+                       List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   
+--------+--------+-------------------+-----------+--------------+---------
+ public | stts_1 | a, b FROM stts_t1 | built     |              | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined
+(4 rows)
+
+\dX *stts_hoge
+                               List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   
+--------+-----------+-------------------------------+-----------+--------------+---------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined
+(1 row)
+
+\dX+
+                                                                   List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+----------+------------------------+--------------------------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        |         |                | 106 bytes         | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built   |                |                   | 24 kB
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built   |                |                   | 386 bytes
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built   |                |                   | 294 bytes
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              |         | 13 bytes       |                   | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined |                | 0 bytes           | 0 bytes
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built   |                |                   | 686 bytes
+(12 rows)
+
+\dX+ stts_?
+                                                List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+--------+--------+-------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public | stts_1 | a, b FROM stts_t1 | built     |              |         | 13 bytes       |                   | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(4 rows)
+
+\dX+ *stts_hoge
+                                                       List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+--------+-----------+-------------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(1 row)
+
+\dX+ stts_s2.stts_yama
+                                                    List of extended statistics
+ Schema  |   Name    |       Definition        | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+---------+-----------+-------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ stts_s2 | stts_yama | col1, col3 FROM stts_t3 |           | defined      | defined |                | 0 bytes           | 0 bytes
+(1 row)
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 7912e733ae..db6e3e1ba3 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -912,6 +912,37 @@ CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
 
 ANALYZE tststats.priv_test_tbl;
 
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+
+\dX
+\dX stts_?
+\dX *stts_hoge
+\dX+
+\dX+ stts_?
+\dX+ *stts_hoge
+\dX+ stts_s2.stts_yama
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
+
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
-- 
2.26.2


--------------B0278A9E952F6668D2DBEBA1
Content-Type: text/x-patch; charset=UTF-8;
 name="0002-fixup-rename-defined-to-requested-20210109.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="0002-fixup-rename-defined-to-requested-20210109.patch"



^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* [PATCH 1/2] psql \dX: list extended statistics objects
@ 2021-01-07 05:28 Tatsuro Yamada <[email protected]>
  0 siblings, 0 replies; 30+ messages in thread

From: Tatsuro Yamada @ 2021-01-07 05:28 UTC (permalink / raw)

The new command lists extended statistics objects, possibly with their
sizes. All past releases with extended statistics are supported.

Author: Tatsuro Yamada
Reviewed-by: Julien Rouhaud, Alvaro Herrera, Tomas Vondra
Discussion: https://postgr.es/m/c027a541-5856-75a5-0868-341301e1624b%40nttcom.co.jp_1
---
 doc/src/sgml/ref/psql-ref.sgml          |  14 +++
 src/bin/psql/command.c                  |   3 +
 src/bin/psql/describe.c                 | 150 ++++++++++++++++++++++++
 src/bin/psql/describe.h                 |   3 +
 src/bin/psql/help.c                     |   1 +
 src/bin/psql/tab-complete.c             |   4 +-
 src/test/regress/expected/stats_ext.out |  94 +++++++++++++++
 src/test/regress/sql/stats_ext.sql      |  31 +++++
 8 files changed, 299 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml
index 221a967bfe..d01acc92b8 100644
--- a/doc/src/sgml/ref/psql-ref.sgml
+++ b/doc/src/sgml/ref/psql-ref.sgml
@@ -1918,6 +1918,20 @@ testdb=&gt;
         </para>
         </listitem>
       </varlistentry>
+      
+      <varlistentry>
+        <term><literal>\dX [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
+        <listitem>
+        <para>
+        Lists extended statistics.
+        If <replaceable class="parameter">pattern</replaceable>
+        is specified, only those extended statistics whose names match the
+        pattern are listed.
+        If <literal>+</literal> is appended to the command name, each extended
+        statistics is listed with its size.
+        </para>
+        </listitem>
+      </varlistentry>
 
       <varlistentry>
         <term><literal>\dy[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index 303e7c3ad8..c5ebc1c3f4 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -928,6 +928,9 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd)
 				else
 					success = listExtensions(pattern);
 				break;
+			case 'X':			/* Extended Statistics */
+				success = listExtendedStats(pattern, show_verbose);
+				break;
 			case 'y':			/* Event Triggers */
 				success = listEventTriggers(pattern, show_verbose);
 				break;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index caf97563f4..46f54199fb 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -4392,6 +4392,156 @@ listEventTriggers(const char *pattern, bool verbose)
 	return true;
 }
 
+/*
+ * \dX
+ *
+ * Describes extended statistics.
+ */
+bool
+listExtendedStats(const char *pattern, bool verbose)
+{
+	PQExpBufferData buf;
+	PGresult   *res;
+	printQueryOpt myopt = pset.popt;
+
+	if (pset.sversion < 100000)
+	{
+		char		sverbuf[32];
+
+		pg_log_error("The server (version %s) does not support extended statistics.",
+					 formatPGVersionNumber(pset.sversion, false,
+										   sverbuf, sizeof(sverbuf)));
+		return true;
+	}
+
+	initPQExpBuffer(&buf);
+	printfPQExpBuffer(&buf,
+					  "SELECT \n"
+					  "es.stxnamespace::pg_catalog.regnamespace::text AS \"%s\", \n"
+					  "es.stxname AS \"%s\", \n"
+					  "pg_catalog.format('%%s FROM %%s', \n"
+					  "  (SELECT pg_catalog.string_agg(pg_catalog.quote_ident(a.attname),', ') \n"
+					  "   FROM pg_catalog.unnest(es.stxkeys) s(attnum) \n"
+					  "   JOIN pg_catalog.pg_attribute a \n"
+					  "   ON (es.stxrelid = a.attrelid \n"
+					  "   AND a.attnum = s.attnum \n"
+					  "   AND NOT a.attisdropped)), \n"
+					  "es.stxrelid::regclass) AS \"%s\"",
+					  gettext_noop("Schema"),
+					  gettext_noop("Name"),
+					  gettext_noop("Definition"));
+
+	/*
+	 * Since 12 there are two catalogs - one for the definition, one for the
+	 * data built by ANALYZE. Older releases use a single catalog. Also, 12
+	 * adds the MCV statistics kind.
+	 */
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN es.stxdependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"));
+	}
+	else
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxddependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxdmcv IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'm' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"),
+						  gettext_noop("MCV"));
+	}
+
+	/* In verbose mode, print sizes of the extended statistics objects. */
+	if (verbose)
+	{
+		if (pset.sversion < 120000)
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN es.stxdependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxdependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"));
+		}
+		else
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxddependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxddependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxdmcv IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdmcv)::bigint) \n"
+							  "     WHEN 'm' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"),
+							  gettext_noop("MCV_size"));
+		}
+	}
+
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+	else
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "LEFT JOIN pg_catalog.pg_statistic_ext_data esd \n"
+							 "ON es.oid = esd.stxoid \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+
+	processSQLNamePattern(pset.db, &buf, pattern, false,
+						  false, NULL,
+						  "stxname", NULL,
+						  NULL);
+
+	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
+
+	res = PSQLexec(buf.data);
+	termPQExpBuffer(&buf);
+	if (!res)
+		return false;
+
+	myopt.nullPrint = NULL;
+	myopt.title = _("List of extended statistics");
+	myopt.translate_header = true;
+
+	printQuery(res, &myopt, pset.queryFout, false, pset.logfile);
+
+	PQclear(res);
+	return true;
+}
+
 /*
  * \dC
  *
diff --git a/src/bin/psql/describe.h b/src/bin/psql/describe.h
index 6044e3a082..867e57d851 100644
--- a/src/bin/psql/describe.h
+++ b/src/bin/psql/describe.h
@@ -102,6 +102,9 @@ extern bool listExtensions(const char *pattern);
 /* \dx+ */
 extern bool listExtensionContents(const char *pattern);
 
+/* \dX */
+extern bool listExtendedStats(const char *pattern, bool verbose);
+
 /* \dy */
 extern bool listEventTriggers(const char *pattern, bool verbose);
 
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 9ec1c4e810..e42bc8c54e 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -267,6 +267,7 @@ slashUsage(unsigned short int pager)
 	fprintf(output, _("  \\du[S+] [PATTERN]      list roles\n"));
 	fprintf(output, _("  \\dv[S+] [PATTERN]      list views\n"));
 	fprintf(output, _("  \\dx[+]  [PATTERN]      list extensions\n"));
+	fprintf(output, _("  \\dX[+]  [PATTERN]      list extended statistics\n"));
 	fprintf(output, _("  \\dy     [PATTERN]      list event triggers\n"));
 	fprintf(output, _("  \\l[+]   [PATTERN]      list databases\n"));
 	fprintf(output, _("  \\sf[+]  FUNCNAME       show a function's definition\n"));
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9dcab0d2fa..611f1efb15 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1500,7 +1500,7 @@ psql_completion(const char *text, int start, int end)
 		"\\dF", "\\dFd", "\\dFp", "\\dFt", "\\dg", "\\di", "\\dl", "\\dL",
 		"\\dm", "\\dn", "\\do", "\\dO", "\\dp", "\\dP", "\\dPi", "\\dPt",
 		"\\drds", "\\dRs", "\\dRp", "\\ds", "\\dS",
-		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dy",
+		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dX", "\\dy",
 		"\\e", "\\echo", "\\ef", "\\elif", "\\else", "\\encoding",
 		"\\endif", "\\errverbose", "\\ev",
 		"\\f",
@@ -3910,6 +3910,8 @@ psql_completion(const char *text, int start, int end)
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_views, NULL);
 	else if (TailMatchesCS("\\dx*"))
 		COMPLETE_WITH_QUERY(Query_for_list_of_extensions);
+	else if (TailMatchesCS("\\dX*"))
+		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_statistics, NULL);
 	else if (TailMatchesCS("\\dm*"))
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_matviews, NULL);
 	else if (TailMatchesCS("\\dE*"))
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 7bfeaf85f0..8c8a0afcf6 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -1725,6 +1725,100 @@ INSERT INTO tststats.priv_test_tbl
 CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
   FROM tststats.priv_test_tbl;
 ANALYZE tststats.priv_test_tbl;
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+\dX
+                                          List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   
+----------+------------------------+--------------------------------------+-----------+--------------+---------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built
+(12 rows)
+
+\dX stts_?
+                       List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   
+--------+--------+-------------------+-----------+--------------+---------
+ public | stts_1 | a, b FROM stts_t1 | built     |              | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined
+(4 rows)
+
+\dX *stts_hoge
+                               List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   
+--------+-----------+-------------------------------+-----------+--------------+---------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined
+(1 row)
+
+\dX+
+                                                                   List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+----------+------------------------+--------------------------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        |         |                | 106 bytes         | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built   |                |                   | 24 kB
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built   |                |                   | 386 bytes
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built   |                |                   | 294 bytes
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              |         | 13 bytes       |                   | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined |                | 0 bytes           | 0 bytes
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built   |                |                   | 686 bytes
+(12 rows)
+
+\dX+ stts_?
+                                                List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+--------+--------+-------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public | stts_1 | a, b FROM stts_t1 | built     |              |         | 13 bytes       |                   | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(4 rows)
+
+\dX+ *stts_hoge
+                                                       List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+--------+-----------+-------------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(1 row)
+
+\dX+ stts_s2.stts_yama
+                                                    List of extended statistics
+ Schema  |   Name    |       Definition        | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+---------+-----------+-------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ stts_s2 | stts_yama | col1, col3 FROM stts_t3 |           | defined      | defined |                | 0 bytes           | 0 bytes
+(1 row)
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 7912e733ae..db6e3e1ba3 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -912,6 +912,37 @@ CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
 
 ANALYZE tststats.priv_test_tbl;
 
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+
+\dX
+\dX stts_?
+\dX *stts_hoge
+\dX+
+\dX+ stts_?
+\dX+ *stts_hoge
+\dX+ stts_s2.stts_yama
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
+
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
-- 
2.26.2


--------------B0278A9E952F6668D2DBEBA1
Content-Type: text/x-patch; charset=UTF-8;
 name="0002-fixup-rename-defined-to-requested-20210109.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="0002-fixup-rename-defined-to-requested-20210109.patch"



^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* [PATCH 1/2] psql \dX: list extended statistics objects
@ 2021-01-07 05:28 Tatsuro Yamada <[email protected]>
  0 siblings, 0 replies; 30+ messages in thread

From: Tatsuro Yamada @ 2021-01-07 05:28 UTC (permalink / raw)

The new command lists extended statistics objects, possibly with their
sizes. All past releases with extended statistics are supported.

Author: Tatsuro Yamada
Reviewed-by: Julien Rouhaud, Alvaro Herrera, Tomas Vondra
Discussion: https://postgr.es/m/c027a541-5856-75a5-0868-341301e1624b%40nttcom.co.jp_1
---
 doc/src/sgml/ref/psql-ref.sgml          |  14 +++
 src/bin/psql/command.c                  |   3 +
 src/bin/psql/describe.c                 | 150 ++++++++++++++++++++++++
 src/bin/psql/describe.h                 |   3 +
 src/bin/psql/help.c                     |   1 +
 src/bin/psql/tab-complete.c             |   4 +-
 src/test/regress/expected/stats_ext.out |  94 +++++++++++++++
 src/test/regress/sql/stats_ext.sql      |  31 +++++
 8 files changed, 299 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml
index 221a967bfe..d01acc92b8 100644
--- a/doc/src/sgml/ref/psql-ref.sgml
+++ b/doc/src/sgml/ref/psql-ref.sgml
@@ -1918,6 +1918,20 @@ testdb=&gt;
         </para>
         </listitem>
       </varlistentry>
+      
+      <varlistentry>
+        <term><literal>\dX [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
+        <listitem>
+        <para>
+        Lists extended statistics.
+        If <replaceable class="parameter">pattern</replaceable>
+        is specified, only those extended statistics whose names match the
+        pattern are listed.
+        If <literal>+</literal> is appended to the command name, each extended
+        statistics is listed with its size.
+        </para>
+        </listitem>
+      </varlistentry>
 
       <varlistentry>
         <term><literal>\dy[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index 303e7c3ad8..c5ebc1c3f4 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -928,6 +928,9 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd)
 				else
 					success = listExtensions(pattern);
 				break;
+			case 'X':			/* Extended Statistics */
+				success = listExtendedStats(pattern, show_verbose);
+				break;
 			case 'y':			/* Event Triggers */
 				success = listEventTriggers(pattern, show_verbose);
 				break;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index caf97563f4..46f54199fb 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -4392,6 +4392,156 @@ listEventTriggers(const char *pattern, bool verbose)
 	return true;
 }
 
+/*
+ * \dX
+ *
+ * Describes extended statistics.
+ */
+bool
+listExtendedStats(const char *pattern, bool verbose)
+{
+	PQExpBufferData buf;
+	PGresult   *res;
+	printQueryOpt myopt = pset.popt;
+
+	if (pset.sversion < 100000)
+	{
+		char		sverbuf[32];
+
+		pg_log_error("The server (version %s) does not support extended statistics.",
+					 formatPGVersionNumber(pset.sversion, false,
+										   sverbuf, sizeof(sverbuf)));
+		return true;
+	}
+
+	initPQExpBuffer(&buf);
+	printfPQExpBuffer(&buf,
+					  "SELECT \n"
+					  "es.stxnamespace::pg_catalog.regnamespace::text AS \"%s\", \n"
+					  "es.stxname AS \"%s\", \n"
+					  "pg_catalog.format('%%s FROM %%s', \n"
+					  "  (SELECT pg_catalog.string_agg(pg_catalog.quote_ident(a.attname),', ') \n"
+					  "   FROM pg_catalog.unnest(es.stxkeys) s(attnum) \n"
+					  "   JOIN pg_catalog.pg_attribute a \n"
+					  "   ON (es.stxrelid = a.attrelid \n"
+					  "   AND a.attnum = s.attnum \n"
+					  "   AND NOT a.attisdropped)), \n"
+					  "es.stxrelid::regclass) AS \"%s\"",
+					  gettext_noop("Schema"),
+					  gettext_noop("Name"),
+					  gettext_noop("Definition"));
+
+	/*
+	 * Since 12 there are two catalogs - one for the definition, one for the
+	 * data built by ANALYZE. Older releases use a single catalog. Also, 12
+	 * adds the MCV statistics kind.
+	 */
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN es.stxdependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"));
+	}
+	else
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxddependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxdmcv IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'm' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"),
+						  gettext_noop("MCV"));
+	}
+
+	/* In verbose mode, print sizes of the extended statistics objects. */
+	if (verbose)
+	{
+		if (pset.sversion < 120000)
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN es.stxdependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxdependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"));
+		}
+		else
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxddependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxddependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxdmcv IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdmcv)::bigint) \n"
+							  "     WHEN 'm' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"),
+							  gettext_noop("MCV_size"));
+		}
+	}
+
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+	else
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "LEFT JOIN pg_catalog.pg_statistic_ext_data esd \n"
+							 "ON es.oid = esd.stxoid \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+
+	processSQLNamePattern(pset.db, &buf, pattern, false,
+						  false, NULL,
+						  "stxname", NULL,
+						  NULL);
+
+	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
+
+	res = PSQLexec(buf.data);
+	termPQExpBuffer(&buf);
+	if (!res)
+		return false;
+
+	myopt.nullPrint = NULL;
+	myopt.title = _("List of extended statistics");
+	myopt.translate_header = true;
+
+	printQuery(res, &myopt, pset.queryFout, false, pset.logfile);
+
+	PQclear(res);
+	return true;
+}
+
 /*
  * \dC
  *
diff --git a/src/bin/psql/describe.h b/src/bin/psql/describe.h
index 6044e3a082..867e57d851 100644
--- a/src/bin/psql/describe.h
+++ b/src/bin/psql/describe.h
@@ -102,6 +102,9 @@ extern bool listExtensions(const char *pattern);
 /* \dx+ */
 extern bool listExtensionContents(const char *pattern);
 
+/* \dX */
+extern bool listExtendedStats(const char *pattern, bool verbose);
+
 /* \dy */
 extern bool listEventTriggers(const char *pattern, bool verbose);
 
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 9ec1c4e810..e42bc8c54e 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -267,6 +267,7 @@ slashUsage(unsigned short int pager)
 	fprintf(output, _("  \\du[S+] [PATTERN]      list roles\n"));
 	fprintf(output, _("  \\dv[S+] [PATTERN]      list views\n"));
 	fprintf(output, _("  \\dx[+]  [PATTERN]      list extensions\n"));
+	fprintf(output, _("  \\dX[+]  [PATTERN]      list extended statistics\n"));
 	fprintf(output, _("  \\dy     [PATTERN]      list event triggers\n"));
 	fprintf(output, _("  \\l[+]   [PATTERN]      list databases\n"));
 	fprintf(output, _("  \\sf[+]  FUNCNAME       show a function's definition\n"));
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9dcab0d2fa..611f1efb15 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1500,7 +1500,7 @@ psql_completion(const char *text, int start, int end)
 		"\\dF", "\\dFd", "\\dFp", "\\dFt", "\\dg", "\\di", "\\dl", "\\dL",
 		"\\dm", "\\dn", "\\do", "\\dO", "\\dp", "\\dP", "\\dPi", "\\dPt",
 		"\\drds", "\\dRs", "\\dRp", "\\ds", "\\dS",
-		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dy",
+		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dX", "\\dy",
 		"\\e", "\\echo", "\\ef", "\\elif", "\\else", "\\encoding",
 		"\\endif", "\\errverbose", "\\ev",
 		"\\f",
@@ -3910,6 +3910,8 @@ psql_completion(const char *text, int start, int end)
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_views, NULL);
 	else if (TailMatchesCS("\\dx*"))
 		COMPLETE_WITH_QUERY(Query_for_list_of_extensions);
+	else if (TailMatchesCS("\\dX*"))
+		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_statistics, NULL);
 	else if (TailMatchesCS("\\dm*"))
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_matviews, NULL);
 	else if (TailMatchesCS("\\dE*"))
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 7bfeaf85f0..8c8a0afcf6 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -1725,6 +1725,100 @@ INSERT INTO tststats.priv_test_tbl
 CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
   FROM tststats.priv_test_tbl;
 ANALYZE tststats.priv_test_tbl;
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+\dX
+                                          List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   
+----------+------------------------+--------------------------------------+-----------+--------------+---------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built
+(12 rows)
+
+\dX stts_?
+                       List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   
+--------+--------+-------------------+-----------+--------------+---------
+ public | stts_1 | a, b FROM stts_t1 | built     |              | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined
+(4 rows)
+
+\dX *stts_hoge
+                               List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   
+--------+-----------+-------------------------------+-----------+--------------+---------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined
+(1 row)
+
+\dX+
+                                                                   List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+----------+------------------------+--------------------------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        |         |                | 106 bytes         | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built   |                |                   | 24 kB
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built   |                |                   | 386 bytes
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built   |                |                   | 294 bytes
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              |         | 13 bytes       |                   | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined |                | 0 bytes           | 0 bytes
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built   |                |                   | 686 bytes
+(12 rows)
+
+\dX+ stts_?
+                                                List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+--------+--------+-------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public | stts_1 | a, b FROM stts_t1 | built     |              |         | 13 bytes       |                   | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(4 rows)
+
+\dX+ *stts_hoge
+                                                       List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+--------+-----------+-------------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(1 row)
+
+\dX+ stts_s2.stts_yama
+                                                    List of extended statistics
+ Schema  |   Name    |       Definition        | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+---------+-----------+-------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ stts_s2 | stts_yama | col1, col3 FROM stts_t3 |           | defined      | defined |                | 0 bytes           | 0 bytes
+(1 row)
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 7912e733ae..db6e3e1ba3 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -912,6 +912,37 @@ CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
 
 ANALYZE tststats.priv_test_tbl;
 
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+
+\dX
+\dX stts_?
+\dX *stts_hoge
+\dX+
+\dX+ stts_?
+\dX+ *stts_hoge
+\dX+ stts_s2.stts_yama
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
+
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
-- 
2.26.2


--------------B0278A9E952F6668D2DBEBA1
Content-Type: text/x-patch; charset=UTF-8;
 name="0002-fixup-rename-defined-to-requested-20210109.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="0002-fixup-rename-defined-to-requested-20210109.patch"



^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* [PATCH 1/2] psql \dX: list extended statistics objects
@ 2021-01-07 05:28 Tatsuro Yamada <[email protected]>
  0 siblings, 0 replies; 30+ messages in thread

From: Tatsuro Yamada @ 2021-01-07 05:28 UTC (permalink / raw)

The new command lists extended statistics objects, possibly with their
sizes. All past releases with extended statistics are supported.

Author: Tatsuro Yamada
Reviewed-by: Julien Rouhaud, Alvaro Herrera, Tomas Vondra
Discussion: https://postgr.es/m/c027a541-5856-75a5-0868-341301e1624b%40nttcom.co.jp_1
---
 doc/src/sgml/ref/psql-ref.sgml          |  14 +++
 src/bin/psql/command.c                  |   3 +
 src/bin/psql/describe.c                 | 150 ++++++++++++++++++++++++
 src/bin/psql/describe.h                 |   3 +
 src/bin/psql/help.c                     |   1 +
 src/bin/psql/tab-complete.c             |   4 +-
 src/test/regress/expected/stats_ext.out |  94 +++++++++++++++
 src/test/regress/sql/stats_ext.sql      |  31 +++++
 8 files changed, 299 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml
index 221a967bfe..d01acc92b8 100644
--- a/doc/src/sgml/ref/psql-ref.sgml
+++ b/doc/src/sgml/ref/psql-ref.sgml
@@ -1918,6 +1918,20 @@ testdb=&gt;
         </para>
         </listitem>
       </varlistentry>
+      
+      <varlistentry>
+        <term><literal>\dX [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
+        <listitem>
+        <para>
+        Lists extended statistics.
+        If <replaceable class="parameter">pattern</replaceable>
+        is specified, only those extended statistics whose names match the
+        pattern are listed.
+        If <literal>+</literal> is appended to the command name, each extended
+        statistics is listed with its size.
+        </para>
+        </listitem>
+      </varlistentry>
 
       <varlistentry>
         <term><literal>\dy[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index 303e7c3ad8..c5ebc1c3f4 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -928,6 +928,9 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd)
 				else
 					success = listExtensions(pattern);
 				break;
+			case 'X':			/* Extended Statistics */
+				success = listExtendedStats(pattern, show_verbose);
+				break;
 			case 'y':			/* Event Triggers */
 				success = listEventTriggers(pattern, show_verbose);
 				break;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index caf97563f4..46f54199fb 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -4392,6 +4392,156 @@ listEventTriggers(const char *pattern, bool verbose)
 	return true;
 }
 
+/*
+ * \dX
+ *
+ * Describes extended statistics.
+ */
+bool
+listExtendedStats(const char *pattern, bool verbose)
+{
+	PQExpBufferData buf;
+	PGresult   *res;
+	printQueryOpt myopt = pset.popt;
+
+	if (pset.sversion < 100000)
+	{
+		char		sverbuf[32];
+
+		pg_log_error("The server (version %s) does not support extended statistics.",
+					 formatPGVersionNumber(pset.sversion, false,
+										   sverbuf, sizeof(sverbuf)));
+		return true;
+	}
+
+	initPQExpBuffer(&buf);
+	printfPQExpBuffer(&buf,
+					  "SELECT \n"
+					  "es.stxnamespace::pg_catalog.regnamespace::text AS \"%s\", \n"
+					  "es.stxname AS \"%s\", \n"
+					  "pg_catalog.format('%%s FROM %%s', \n"
+					  "  (SELECT pg_catalog.string_agg(pg_catalog.quote_ident(a.attname),', ') \n"
+					  "   FROM pg_catalog.unnest(es.stxkeys) s(attnum) \n"
+					  "   JOIN pg_catalog.pg_attribute a \n"
+					  "   ON (es.stxrelid = a.attrelid \n"
+					  "   AND a.attnum = s.attnum \n"
+					  "   AND NOT a.attisdropped)), \n"
+					  "es.stxrelid::regclass) AS \"%s\"",
+					  gettext_noop("Schema"),
+					  gettext_noop("Name"),
+					  gettext_noop("Definition"));
+
+	/*
+	 * Since 12 there are two catalogs - one for the definition, one for the
+	 * data built by ANALYZE. Older releases use a single catalog. Also, 12
+	 * adds the MCV statistics kind.
+	 */
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN es.stxdependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"));
+	}
+	else
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxddependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxdmcv IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'm' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"),
+						  gettext_noop("MCV"));
+	}
+
+	/* In verbose mode, print sizes of the extended statistics objects. */
+	if (verbose)
+	{
+		if (pset.sversion < 120000)
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN es.stxdependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxdependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"));
+		}
+		else
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxddependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxddependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxdmcv IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdmcv)::bigint) \n"
+							  "     WHEN 'm' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"),
+							  gettext_noop("MCV_size"));
+		}
+	}
+
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+	else
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "LEFT JOIN pg_catalog.pg_statistic_ext_data esd \n"
+							 "ON es.oid = esd.stxoid \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+
+	processSQLNamePattern(pset.db, &buf, pattern, false,
+						  false, NULL,
+						  "stxname", NULL,
+						  NULL);
+
+	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
+
+	res = PSQLexec(buf.data);
+	termPQExpBuffer(&buf);
+	if (!res)
+		return false;
+
+	myopt.nullPrint = NULL;
+	myopt.title = _("List of extended statistics");
+	myopt.translate_header = true;
+
+	printQuery(res, &myopt, pset.queryFout, false, pset.logfile);
+
+	PQclear(res);
+	return true;
+}
+
 /*
  * \dC
  *
diff --git a/src/bin/psql/describe.h b/src/bin/psql/describe.h
index 6044e3a082..867e57d851 100644
--- a/src/bin/psql/describe.h
+++ b/src/bin/psql/describe.h
@@ -102,6 +102,9 @@ extern bool listExtensions(const char *pattern);
 /* \dx+ */
 extern bool listExtensionContents(const char *pattern);
 
+/* \dX */
+extern bool listExtendedStats(const char *pattern, bool verbose);
+
 /* \dy */
 extern bool listEventTriggers(const char *pattern, bool verbose);
 
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 9ec1c4e810..e42bc8c54e 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -267,6 +267,7 @@ slashUsage(unsigned short int pager)
 	fprintf(output, _("  \\du[S+] [PATTERN]      list roles\n"));
 	fprintf(output, _("  \\dv[S+] [PATTERN]      list views\n"));
 	fprintf(output, _("  \\dx[+]  [PATTERN]      list extensions\n"));
+	fprintf(output, _("  \\dX[+]  [PATTERN]      list extended statistics\n"));
 	fprintf(output, _("  \\dy     [PATTERN]      list event triggers\n"));
 	fprintf(output, _("  \\l[+]   [PATTERN]      list databases\n"));
 	fprintf(output, _("  \\sf[+]  FUNCNAME       show a function's definition\n"));
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9dcab0d2fa..611f1efb15 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1500,7 +1500,7 @@ psql_completion(const char *text, int start, int end)
 		"\\dF", "\\dFd", "\\dFp", "\\dFt", "\\dg", "\\di", "\\dl", "\\dL",
 		"\\dm", "\\dn", "\\do", "\\dO", "\\dp", "\\dP", "\\dPi", "\\dPt",
 		"\\drds", "\\dRs", "\\dRp", "\\ds", "\\dS",
-		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dy",
+		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dX", "\\dy",
 		"\\e", "\\echo", "\\ef", "\\elif", "\\else", "\\encoding",
 		"\\endif", "\\errverbose", "\\ev",
 		"\\f",
@@ -3910,6 +3910,8 @@ psql_completion(const char *text, int start, int end)
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_views, NULL);
 	else if (TailMatchesCS("\\dx*"))
 		COMPLETE_WITH_QUERY(Query_for_list_of_extensions);
+	else if (TailMatchesCS("\\dX*"))
+		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_statistics, NULL);
 	else if (TailMatchesCS("\\dm*"))
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_matviews, NULL);
 	else if (TailMatchesCS("\\dE*"))
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 7bfeaf85f0..8c8a0afcf6 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -1725,6 +1725,100 @@ INSERT INTO tststats.priv_test_tbl
 CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
   FROM tststats.priv_test_tbl;
 ANALYZE tststats.priv_test_tbl;
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+\dX
+                                          List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   
+----------+------------------------+--------------------------------------+-----------+--------------+---------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built
+(12 rows)
+
+\dX stts_?
+                       List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   
+--------+--------+-------------------+-----------+--------------+---------
+ public | stts_1 | a, b FROM stts_t1 | built     |              | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined
+(4 rows)
+
+\dX *stts_hoge
+                               List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   
+--------+-----------+-------------------------------+-----------+--------------+---------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined
+(1 row)
+
+\dX+
+                                                                   List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+----------+------------------------+--------------------------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        |         |                | 106 bytes         | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built   |                |                   | 24 kB
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built   |                |                   | 386 bytes
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built   |                |                   | 294 bytes
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              |         | 13 bytes       |                   | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined |                | 0 bytes           | 0 bytes
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built   |                |                   | 686 bytes
+(12 rows)
+
+\dX+ stts_?
+                                                List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+--------+--------+-------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public | stts_1 | a, b FROM stts_t1 | built     |              |         | 13 bytes       |                   | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(4 rows)
+
+\dX+ *stts_hoge
+                                                       List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+--------+-----------+-------------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(1 row)
+
+\dX+ stts_s2.stts_yama
+                                                    List of extended statistics
+ Schema  |   Name    |       Definition        | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+---------+-----------+-------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ stts_s2 | stts_yama | col1, col3 FROM stts_t3 |           | defined      | defined |                | 0 bytes           | 0 bytes
+(1 row)
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 7912e733ae..db6e3e1ba3 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -912,6 +912,37 @@ CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
 
 ANALYZE tststats.priv_test_tbl;
 
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+
+\dX
+\dX stts_?
+\dX *stts_hoge
+\dX+
+\dX+ stts_?
+\dX+ *stts_hoge
+\dX+ stts_s2.stts_yama
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
+
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
-- 
2.26.2


--------------B0278A9E952F6668D2DBEBA1
Content-Type: text/x-patch; charset=UTF-8;
 name="0002-fixup-rename-defined-to-requested-20210109.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="0002-fixup-rename-defined-to-requested-20210109.patch"



^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* [PATCH 1/2] psql \dX: list extended statistics objects
@ 2021-01-07 05:28 Tatsuro Yamada <[email protected]>
  0 siblings, 0 replies; 30+ messages in thread

From: Tatsuro Yamada @ 2021-01-07 05:28 UTC (permalink / raw)

The new command lists extended statistics objects, possibly with their
sizes. All past releases with extended statistics are supported.

Author: Tatsuro Yamada
Reviewed-by: Julien Rouhaud, Alvaro Herrera, Tomas Vondra
Discussion: https://postgr.es/m/c027a541-5856-75a5-0868-341301e1624b%40nttcom.co.jp_1
---
 doc/src/sgml/ref/psql-ref.sgml          |  14 +++
 src/bin/psql/command.c                  |   3 +
 src/bin/psql/describe.c                 | 150 ++++++++++++++++++++++++
 src/bin/psql/describe.h                 |   3 +
 src/bin/psql/help.c                     |   1 +
 src/bin/psql/tab-complete.c             |   4 +-
 src/test/regress/expected/stats_ext.out |  94 +++++++++++++++
 src/test/regress/sql/stats_ext.sql      |  31 +++++
 8 files changed, 299 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml
index 221a967bfe..d01acc92b8 100644
--- a/doc/src/sgml/ref/psql-ref.sgml
+++ b/doc/src/sgml/ref/psql-ref.sgml
@@ -1918,6 +1918,20 @@ testdb=&gt;
         </para>
         </listitem>
       </varlistentry>
+      
+      <varlistentry>
+        <term><literal>\dX [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
+        <listitem>
+        <para>
+        Lists extended statistics.
+        If <replaceable class="parameter">pattern</replaceable>
+        is specified, only those extended statistics whose names match the
+        pattern are listed.
+        If <literal>+</literal> is appended to the command name, each extended
+        statistics is listed with its size.
+        </para>
+        </listitem>
+      </varlistentry>
 
       <varlistentry>
         <term><literal>\dy[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index 303e7c3ad8..c5ebc1c3f4 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -928,6 +928,9 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd)
 				else
 					success = listExtensions(pattern);
 				break;
+			case 'X':			/* Extended Statistics */
+				success = listExtendedStats(pattern, show_verbose);
+				break;
 			case 'y':			/* Event Triggers */
 				success = listEventTriggers(pattern, show_verbose);
 				break;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index caf97563f4..46f54199fb 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -4392,6 +4392,156 @@ listEventTriggers(const char *pattern, bool verbose)
 	return true;
 }
 
+/*
+ * \dX
+ *
+ * Describes extended statistics.
+ */
+bool
+listExtendedStats(const char *pattern, bool verbose)
+{
+	PQExpBufferData buf;
+	PGresult   *res;
+	printQueryOpt myopt = pset.popt;
+
+	if (pset.sversion < 100000)
+	{
+		char		sverbuf[32];
+
+		pg_log_error("The server (version %s) does not support extended statistics.",
+					 formatPGVersionNumber(pset.sversion, false,
+										   sverbuf, sizeof(sverbuf)));
+		return true;
+	}
+
+	initPQExpBuffer(&buf);
+	printfPQExpBuffer(&buf,
+					  "SELECT \n"
+					  "es.stxnamespace::pg_catalog.regnamespace::text AS \"%s\", \n"
+					  "es.stxname AS \"%s\", \n"
+					  "pg_catalog.format('%%s FROM %%s', \n"
+					  "  (SELECT pg_catalog.string_agg(pg_catalog.quote_ident(a.attname),', ') \n"
+					  "   FROM pg_catalog.unnest(es.stxkeys) s(attnum) \n"
+					  "   JOIN pg_catalog.pg_attribute a \n"
+					  "   ON (es.stxrelid = a.attrelid \n"
+					  "   AND a.attnum = s.attnum \n"
+					  "   AND NOT a.attisdropped)), \n"
+					  "es.stxrelid::regclass) AS \"%s\"",
+					  gettext_noop("Schema"),
+					  gettext_noop("Name"),
+					  gettext_noop("Definition"));
+
+	/*
+	 * Since 12 there are two catalogs - one for the definition, one for the
+	 * data built by ANALYZE. Older releases use a single catalog. Also, 12
+	 * adds the MCV statistics kind.
+	 */
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN es.stxdependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"));
+	}
+	else
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxddependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxdmcv IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'm' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"),
+						  gettext_noop("MCV"));
+	}
+
+	/* In verbose mode, print sizes of the extended statistics objects. */
+	if (verbose)
+	{
+		if (pset.sversion < 120000)
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN es.stxdependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxdependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"));
+		}
+		else
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxddependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxddependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxdmcv IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdmcv)::bigint) \n"
+							  "     WHEN 'm' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"),
+							  gettext_noop("MCV_size"));
+		}
+	}
+
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+	else
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "LEFT JOIN pg_catalog.pg_statistic_ext_data esd \n"
+							 "ON es.oid = esd.stxoid \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+
+	processSQLNamePattern(pset.db, &buf, pattern, false,
+						  false, NULL,
+						  "stxname", NULL,
+						  NULL);
+
+	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
+
+	res = PSQLexec(buf.data);
+	termPQExpBuffer(&buf);
+	if (!res)
+		return false;
+
+	myopt.nullPrint = NULL;
+	myopt.title = _("List of extended statistics");
+	myopt.translate_header = true;
+
+	printQuery(res, &myopt, pset.queryFout, false, pset.logfile);
+
+	PQclear(res);
+	return true;
+}
+
 /*
  * \dC
  *
diff --git a/src/bin/psql/describe.h b/src/bin/psql/describe.h
index 6044e3a082..867e57d851 100644
--- a/src/bin/psql/describe.h
+++ b/src/bin/psql/describe.h
@@ -102,6 +102,9 @@ extern bool listExtensions(const char *pattern);
 /* \dx+ */
 extern bool listExtensionContents(const char *pattern);
 
+/* \dX */
+extern bool listExtendedStats(const char *pattern, bool verbose);
+
 /* \dy */
 extern bool listEventTriggers(const char *pattern, bool verbose);
 
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 9ec1c4e810..e42bc8c54e 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -267,6 +267,7 @@ slashUsage(unsigned short int pager)
 	fprintf(output, _("  \\du[S+] [PATTERN]      list roles\n"));
 	fprintf(output, _("  \\dv[S+] [PATTERN]      list views\n"));
 	fprintf(output, _("  \\dx[+]  [PATTERN]      list extensions\n"));
+	fprintf(output, _("  \\dX[+]  [PATTERN]      list extended statistics\n"));
 	fprintf(output, _("  \\dy     [PATTERN]      list event triggers\n"));
 	fprintf(output, _("  \\l[+]   [PATTERN]      list databases\n"));
 	fprintf(output, _("  \\sf[+]  FUNCNAME       show a function's definition\n"));
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9dcab0d2fa..611f1efb15 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1500,7 +1500,7 @@ psql_completion(const char *text, int start, int end)
 		"\\dF", "\\dFd", "\\dFp", "\\dFt", "\\dg", "\\di", "\\dl", "\\dL",
 		"\\dm", "\\dn", "\\do", "\\dO", "\\dp", "\\dP", "\\dPi", "\\dPt",
 		"\\drds", "\\dRs", "\\dRp", "\\ds", "\\dS",
-		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dy",
+		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dX", "\\dy",
 		"\\e", "\\echo", "\\ef", "\\elif", "\\else", "\\encoding",
 		"\\endif", "\\errverbose", "\\ev",
 		"\\f",
@@ -3910,6 +3910,8 @@ psql_completion(const char *text, int start, int end)
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_views, NULL);
 	else if (TailMatchesCS("\\dx*"))
 		COMPLETE_WITH_QUERY(Query_for_list_of_extensions);
+	else if (TailMatchesCS("\\dX*"))
+		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_statistics, NULL);
 	else if (TailMatchesCS("\\dm*"))
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_matviews, NULL);
 	else if (TailMatchesCS("\\dE*"))
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 7bfeaf85f0..8c8a0afcf6 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -1725,6 +1725,100 @@ INSERT INTO tststats.priv_test_tbl
 CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
   FROM tststats.priv_test_tbl;
 ANALYZE tststats.priv_test_tbl;
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+\dX
+                                          List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   
+----------+------------------------+--------------------------------------+-----------+--------------+---------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built
+(12 rows)
+
+\dX stts_?
+                       List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   
+--------+--------+-------------------+-----------+--------------+---------
+ public | stts_1 | a, b FROM stts_t1 | built     |              | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined
+(4 rows)
+
+\dX *stts_hoge
+                               List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   
+--------+-----------+-------------------------------+-----------+--------------+---------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined
+(1 row)
+
+\dX+
+                                                                   List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+----------+------------------------+--------------------------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        |         |                | 106 bytes         | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built   |                |                   | 24 kB
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built   |                |                   | 386 bytes
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built   |                |                   | 294 bytes
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              |         | 13 bytes       |                   | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined |                | 0 bytes           | 0 bytes
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built   |                |                   | 686 bytes
+(12 rows)
+
+\dX+ stts_?
+                                                List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+--------+--------+-------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public | stts_1 | a, b FROM stts_t1 | built     |              |         | 13 bytes       |                   | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(4 rows)
+
+\dX+ *stts_hoge
+                                                       List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+--------+-----------+-------------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(1 row)
+
+\dX+ stts_s2.stts_yama
+                                                    List of extended statistics
+ Schema  |   Name    |       Definition        | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+---------+-----------+-------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ stts_s2 | stts_yama | col1, col3 FROM stts_t3 |           | defined      | defined |                | 0 bytes           | 0 bytes
+(1 row)
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 7912e733ae..db6e3e1ba3 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -912,6 +912,37 @@ CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
 
 ANALYZE tststats.priv_test_tbl;
 
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+
+\dX
+\dX stts_?
+\dX *stts_hoge
+\dX+
+\dX+ stts_?
+\dX+ *stts_hoge
+\dX+ stts_s2.stts_yama
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
+
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
-- 
2.26.2


--------------B0278A9E952F6668D2DBEBA1
Content-Type: text/x-patch; charset=UTF-8;
 name="0002-fixup-rename-defined-to-requested-20210109.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="0002-fixup-rename-defined-to-requested-20210109.patch"



^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* [PATCH 1/2] psql \dX: list extended statistics objects
@ 2021-01-07 05:28 Tatsuro Yamada <[email protected]>
  0 siblings, 0 replies; 30+ messages in thread

From: Tatsuro Yamada @ 2021-01-07 05:28 UTC (permalink / raw)

The new command lists extended statistics objects, possibly with their
sizes. All past releases with extended statistics are supported.

Author: Tatsuro Yamada
Reviewed-by: Julien Rouhaud, Alvaro Herrera, Tomas Vondra
Discussion: https://postgr.es/m/c027a541-5856-75a5-0868-341301e1624b%40nttcom.co.jp_1
---
 doc/src/sgml/ref/psql-ref.sgml          |  14 +++
 src/bin/psql/command.c                  |   3 +
 src/bin/psql/describe.c                 | 150 ++++++++++++++++++++++++
 src/bin/psql/describe.h                 |   3 +
 src/bin/psql/help.c                     |   1 +
 src/bin/psql/tab-complete.c             |   4 +-
 src/test/regress/expected/stats_ext.out |  94 +++++++++++++++
 src/test/regress/sql/stats_ext.sql      |  31 +++++
 8 files changed, 299 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml
index 221a967bfe..d01acc92b8 100644
--- a/doc/src/sgml/ref/psql-ref.sgml
+++ b/doc/src/sgml/ref/psql-ref.sgml
@@ -1918,6 +1918,20 @@ testdb=&gt;
         </para>
         </listitem>
       </varlistentry>
+      
+      <varlistentry>
+        <term><literal>\dX [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
+        <listitem>
+        <para>
+        Lists extended statistics.
+        If <replaceable class="parameter">pattern</replaceable>
+        is specified, only those extended statistics whose names match the
+        pattern are listed.
+        If <literal>+</literal> is appended to the command name, each extended
+        statistics is listed with its size.
+        </para>
+        </listitem>
+      </varlistentry>
 
       <varlistentry>
         <term><literal>\dy[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index 303e7c3ad8..c5ebc1c3f4 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -928,6 +928,9 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd)
 				else
 					success = listExtensions(pattern);
 				break;
+			case 'X':			/* Extended Statistics */
+				success = listExtendedStats(pattern, show_verbose);
+				break;
 			case 'y':			/* Event Triggers */
 				success = listEventTriggers(pattern, show_verbose);
 				break;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index caf97563f4..46f54199fb 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -4392,6 +4392,156 @@ listEventTriggers(const char *pattern, bool verbose)
 	return true;
 }
 
+/*
+ * \dX
+ *
+ * Describes extended statistics.
+ */
+bool
+listExtendedStats(const char *pattern, bool verbose)
+{
+	PQExpBufferData buf;
+	PGresult   *res;
+	printQueryOpt myopt = pset.popt;
+
+	if (pset.sversion < 100000)
+	{
+		char		sverbuf[32];
+
+		pg_log_error("The server (version %s) does not support extended statistics.",
+					 formatPGVersionNumber(pset.sversion, false,
+										   sverbuf, sizeof(sverbuf)));
+		return true;
+	}
+
+	initPQExpBuffer(&buf);
+	printfPQExpBuffer(&buf,
+					  "SELECT \n"
+					  "es.stxnamespace::pg_catalog.regnamespace::text AS \"%s\", \n"
+					  "es.stxname AS \"%s\", \n"
+					  "pg_catalog.format('%%s FROM %%s', \n"
+					  "  (SELECT pg_catalog.string_agg(pg_catalog.quote_ident(a.attname),', ') \n"
+					  "   FROM pg_catalog.unnest(es.stxkeys) s(attnum) \n"
+					  "   JOIN pg_catalog.pg_attribute a \n"
+					  "   ON (es.stxrelid = a.attrelid \n"
+					  "   AND a.attnum = s.attnum \n"
+					  "   AND NOT a.attisdropped)), \n"
+					  "es.stxrelid::regclass) AS \"%s\"",
+					  gettext_noop("Schema"),
+					  gettext_noop("Name"),
+					  gettext_noop("Definition"));
+
+	/*
+	 * Since 12 there are two catalogs - one for the definition, one for the
+	 * data built by ANALYZE. Older releases use a single catalog. Also, 12
+	 * adds the MCV statistics kind.
+	 */
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN es.stxdependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"));
+	}
+	else
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxddependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxdmcv IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'm' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"),
+						  gettext_noop("MCV"));
+	}
+
+	/* In verbose mode, print sizes of the extended statistics objects. */
+	if (verbose)
+	{
+		if (pset.sversion < 120000)
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN es.stxdependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxdependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"));
+		}
+		else
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxddependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxddependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxdmcv IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdmcv)::bigint) \n"
+							  "     WHEN 'm' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"),
+							  gettext_noop("MCV_size"));
+		}
+	}
+
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+	else
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "LEFT JOIN pg_catalog.pg_statistic_ext_data esd \n"
+							 "ON es.oid = esd.stxoid \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+
+	processSQLNamePattern(pset.db, &buf, pattern, false,
+						  false, NULL,
+						  "stxname", NULL,
+						  NULL);
+
+	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
+
+	res = PSQLexec(buf.data);
+	termPQExpBuffer(&buf);
+	if (!res)
+		return false;
+
+	myopt.nullPrint = NULL;
+	myopt.title = _("List of extended statistics");
+	myopt.translate_header = true;
+
+	printQuery(res, &myopt, pset.queryFout, false, pset.logfile);
+
+	PQclear(res);
+	return true;
+}
+
 /*
  * \dC
  *
diff --git a/src/bin/psql/describe.h b/src/bin/psql/describe.h
index 6044e3a082..867e57d851 100644
--- a/src/bin/psql/describe.h
+++ b/src/bin/psql/describe.h
@@ -102,6 +102,9 @@ extern bool listExtensions(const char *pattern);
 /* \dx+ */
 extern bool listExtensionContents(const char *pattern);
 
+/* \dX */
+extern bool listExtendedStats(const char *pattern, bool verbose);
+
 /* \dy */
 extern bool listEventTriggers(const char *pattern, bool verbose);
 
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 9ec1c4e810..e42bc8c54e 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -267,6 +267,7 @@ slashUsage(unsigned short int pager)
 	fprintf(output, _("  \\du[S+] [PATTERN]      list roles\n"));
 	fprintf(output, _("  \\dv[S+] [PATTERN]      list views\n"));
 	fprintf(output, _("  \\dx[+]  [PATTERN]      list extensions\n"));
+	fprintf(output, _("  \\dX[+]  [PATTERN]      list extended statistics\n"));
 	fprintf(output, _("  \\dy     [PATTERN]      list event triggers\n"));
 	fprintf(output, _("  \\l[+]   [PATTERN]      list databases\n"));
 	fprintf(output, _("  \\sf[+]  FUNCNAME       show a function's definition\n"));
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9dcab0d2fa..611f1efb15 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1500,7 +1500,7 @@ psql_completion(const char *text, int start, int end)
 		"\\dF", "\\dFd", "\\dFp", "\\dFt", "\\dg", "\\di", "\\dl", "\\dL",
 		"\\dm", "\\dn", "\\do", "\\dO", "\\dp", "\\dP", "\\dPi", "\\dPt",
 		"\\drds", "\\dRs", "\\dRp", "\\ds", "\\dS",
-		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dy",
+		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dX", "\\dy",
 		"\\e", "\\echo", "\\ef", "\\elif", "\\else", "\\encoding",
 		"\\endif", "\\errverbose", "\\ev",
 		"\\f",
@@ -3910,6 +3910,8 @@ psql_completion(const char *text, int start, int end)
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_views, NULL);
 	else if (TailMatchesCS("\\dx*"))
 		COMPLETE_WITH_QUERY(Query_for_list_of_extensions);
+	else if (TailMatchesCS("\\dX*"))
+		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_statistics, NULL);
 	else if (TailMatchesCS("\\dm*"))
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_matviews, NULL);
 	else if (TailMatchesCS("\\dE*"))
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 7bfeaf85f0..8c8a0afcf6 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -1725,6 +1725,100 @@ INSERT INTO tststats.priv_test_tbl
 CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
   FROM tststats.priv_test_tbl;
 ANALYZE tststats.priv_test_tbl;
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+\dX
+                                          List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   
+----------+------------------------+--------------------------------------+-----------+--------------+---------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built
+(12 rows)
+
+\dX stts_?
+                       List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   
+--------+--------+-------------------+-----------+--------------+---------
+ public | stts_1 | a, b FROM stts_t1 | built     |              | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined
+(4 rows)
+
+\dX *stts_hoge
+                               List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   
+--------+-----------+-------------------------------+-----------+--------------+---------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined
+(1 row)
+
+\dX+
+                                                                   List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+----------+------------------------+--------------------------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        |         |                | 106 bytes         | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built   |                |                   | 24 kB
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built   |                |                   | 386 bytes
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built   |                |                   | 294 bytes
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              |         | 13 bytes       |                   | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined |                | 0 bytes           | 0 bytes
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built   |                |                   | 686 bytes
+(12 rows)
+
+\dX+ stts_?
+                                                List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+--------+--------+-------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public | stts_1 | a, b FROM stts_t1 | built     |              |         | 13 bytes       |                   | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(4 rows)
+
+\dX+ *stts_hoge
+                                                       List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+--------+-----------+-------------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(1 row)
+
+\dX+ stts_s2.stts_yama
+                                                    List of extended statistics
+ Schema  |   Name    |       Definition        | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+---------+-----------+-------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ stts_s2 | stts_yama | col1, col3 FROM stts_t3 |           | defined      | defined |                | 0 bytes           | 0 bytes
+(1 row)
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 7912e733ae..db6e3e1ba3 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -912,6 +912,37 @@ CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
 
 ANALYZE tststats.priv_test_tbl;
 
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+
+\dX
+\dX stts_?
+\dX *stts_hoge
+\dX+
+\dX+ stts_?
+\dX+ *stts_hoge
+\dX+ stts_s2.stts_yama
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
+
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
-- 
2.26.2


--------------B0278A9E952F6668D2DBEBA1
Content-Type: text/x-patch; charset=UTF-8;
 name="0002-fixup-rename-defined-to-requested-20210109.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="0002-fixup-rename-defined-to-requested-20210109.patch"



^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* [PATCH 1/2] psql \dX: list extended statistics objects
@ 2021-01-07 05:28 Tatsuro Yamada <[email protected]>
  0 siblings, 0 replies; 30+ messages in thread

From: Tatsuro Yamada @ 2021-01-07 05:28 UTC (permalink / raw)

The new command lists extended statistics objects, possibly with their
sizes. All past releases with extended statistics are supported.

Author: Tatsuro Yamada
Reviewed-by: Julien Rouhaud, Alvaro Herrera, Tomas Vondra
Discussion: https://postgr.es/m/c027a541-5856-75a5-0868-341301e1624b%40nttcom.co.jp_1
---
 doc/src/sgml/ref/psql-ref.sgml          |  14 +++
 src/bin/psql/command.c                  |   3 +
 src/bin/psql/describe.c                 | 150 ++++++++++++++++++++++++
 src/bin/psql/describe.h                 |   3 +
 src/bin/psql/help.c                     |   1 +
 src/bin/psql/tab-complete.c             |   4 +-
 src/test/regress/expected/stats_ext.out |  94 +++++++++++++++
 src/test/regress/sql/stats_ext.sql      |  31 +++++
 8 files changed, 299 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml
index 221a967bfe..d01acc92b8 100644
--- a/doc/src/sgml/ref/psql-ref.sgml
+++ b/doc/src/sgml/ref/psql-ref.sgml
@@ -1918,6 +1918,20 @@ testdb=&gt;
         </para>
         </listitem>
       </varlistentry>
+      
+      <varlistentry>
+        <term><literal>\dX [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
+        <listitem>
+        <para>
+        Lists extended statistics.
+        If <replaceable class="parameter">pattern</replaceable>
+        is specified, only those extended statistics whose names match the
+        pattern are listed.
+        If <literal>+</literal> is appended to the command name, each extended
+        statistics is listed with its size.
+        </para>
+        </listitem>
+      </varlistentry>
 
       <varlistentry>
         <term><literal>\dy[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index 303e7c3ad8..c5ebc1c3f4 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -928,6 +928,9 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd)
 				else
 					success = listExtensions(pattern);
 				break;
+			case 'X':			/* Extended Statistics */
+				success = listExtendedStats(pattern, show_verbose);
+				break;
 			case 'y':			/* Event Triggers */
 				success = listEventTriggers(pattern, show_verbose);
 				break;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index caf97563f4..46f54199fb 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -4392,6 +4392,156 @@ listEventTriggers(const char *pattern, bool verbose)
 	return true;
 }
 
+/*
+ * \dX
+ *
+ * Describes extended statistics.
+ */
+bool
+listExtendedStats(const char *pattern, bool verbose)
+{
+	PQExpBufferData buf;
+	PGresult   *res;
+	printQueryOpt myopt = pset.popt;
+
+	if (pset.sversion < 100000)
+	{
+		char		sverbuf[32];
+
+		pg_log_error("The server (version %s) does not support extended statistics.",
+					 formatPGVersionNumber(pset.sversion, false,
+										   sverbuf, sizeof(sverbuf)));
+		return true;
+	}
+
+	initPQExpBuffer(&buf);
+	printfPQExpBuffer(&buf,
+					  "SELECT \n"
+					  "es.stxnamespace::pg_catalog.regnamespace::text AS \"%s\", \n"
+					  "es.stxname AS \"%s\", \n"
+					  "pg_catalog.format('%%s FROM %%s', \n"
+					  "  (SELECT pg_catalog.string_agg(pg_catalog.quote_ident(a.attname),', ') \n"
+					  "   FROM pg_catalog.unnest(es.stxkeys) s(attnum) \n"
+					  "   JOIN pg_catalog.pg_attribute a \n"
+					  "   ON (es.stxrelid = a.attrelid \n"
+					  "   AND a.attnum = s.attnum \n"
+					  "   AND NOT a.attisdropped)), \n"
+					  "es.stxrelid::regclass) AS \"%s\"",
+					  gettext_noop("Schema"),
+					  gettext_noop("Name"),
+					  gettext_noop("Definition"));
+
+	/*
+	 * Since 12 there are two catalogs - one for the definition, one for the
+	 * data built by ANALYZE. Older releases use a single catalog. Also, 12
+	 * adds the MCV statistics kind.
+	 */
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN es.stxdependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"));
+	}
+	else
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxddependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxdmcv IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'm' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"),
+						  gettext_noop("MCV"));
+	}
+
+	/* In verbose mode, print sizes of the extended statistics objects. */
+	if (verbose)
+	{
+		if (pset.sversion < 120000)
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN es.stxdependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxdependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"));
+		}
+		else
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxddependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxddependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxdmcv IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdmcv)::bigint) \n"
+							  "     WHEN 'm' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"),
+							  gettext_noop("MCV_size"));
+		}
+	}
+
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+	else
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "LEFT JOIN pg_catalog.pg_statistic_ext_data esd \n"
+							 "ON es.oid = esd.stxoid \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+
+	processSQLNamePattern(pset.db, &buf, pattern, false,
+						  false, NULL,
+						  "stxname", NULL,
+						  NULL);
+
+	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
+
+	res = PSQLexec(buf.data);
+	termPQExpBuffer(&buf);
+	if (!res)
+		return false;
+
+	myopt.nullPrint = NULL;
+	myopt.title = _("List of extended statistics");
+	myopt.translate_header = true;
+
+	printQuery(res, &myopt, pset.queryFout, false, pset.logfile);
+
+	PQclear(res);
+	return true;
+}
+
 /*
  * \dC
  *
diff --git a/src/bin/psql/describe.h b/src/bin/psql/describe.h
index 6044e3a082..867e57d851 100644
--- a/src/bin/psql/describe.h
+++ b/src/bin/psql/describe.h
@@ -102,6 +102,9 @@ extern bool listExtensions(const char *pattern);
 /* \dx+ */
 extern bool listExtensionContents(const char *pattern);
 
+/* \dX */
+extern bool listExtendedStats(const char *pattern, bool verbose);
+
 /* \dy */
 extern bool listEventTriggers(const char *pattern, bool verbose);
 
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 9ec1c4e810..e42bc8c54e 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -267,6 +267,7 @@ slashUsage(unsigned short int pager)
 	fprintf(output, _("  \\du[S+] [PATTERN]      list roles\n"));
 	fprintf(output, _("  \\dv[S+] [PATTERN]      list views\n"));
 	fprintf(output, _("  \\dx[+]  [PATTERN]      list extensions\n"));
+	fprintf(output, _("  \\dX[+]  [PATTERN]      list extended statistics\n"));
 	fprintf(output, _("  \\dy     [PATTERN]      list event triggers\n"));
 	fprintf(output, _("  \\l[+]   [PATTERN]      list databases\n"));
 	fprintf(output, _("  \\sf[+]  FUNCNAME       show a function's definition\n"));
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9dcab0d2fa..611f1efb15 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1500,7 +1500,7 @@ psql_completion(const char *text, int start, int end)
 		"\\dF", "\\dFd", "\\dFp", "\\dFt", "\\dg", "\\di", "\\dl", "\\dL",
 		"\\dm", "\\dn", "\\do", "\\dO", "\\dp", "\\dP", "\\dPi", "\\dPt",
 		"\\drds", "\\dRs", "\\dRp", "\\ds", "\\dS",
-		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dy",
+		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dX", "\\dy",
 		"\\e", "\\echo", "\\ef", "\\elif", "\\else", "\\encoding",
 		"\\endif", "\\errverbose", "\\ev",
 		"\\f",
@@ -3910,6 +3910,8 @@ psql_completion(const char *text, int start, int end)
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_views, NULL);
 	else if (TailMatchesCS("\\dx*"))
 		COMPLETE_WITH_QUERY(Query_for_list_of_extensions);
+	else if (TailMatchesCS("\\dX*"))
+		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_statistics, NULL);
 	else if (TailMatchesCS("\\dm*"))
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_matviews, NULL);
 	else if (TailMatchesCS("\\dE*"))
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 7bfeaf85f0..8c8a0afcf6 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -1725,6 +1725,100 @@ INSERT INTO tststats.priv_test_tbl
 CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
   FROM tststats.priv_test_tbl;
 ANALYZE tststats.priv_test_tbl;
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+\dX
+                                          List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   
+----------+------------------------+--------------------------------------+-----------+--------------+---------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built
+(12 rows)
+
+\dX stts_?
+                       List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   
+--------+--------+-------------------+-----------+--------------+---------
+ public | stts_1 | a, b FROM stts_t1 | built     |              | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined
+(4 rows)
+
+\dX *stts_hoge
+                               List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   
+--------+-----------+-------------------------------+-----------+--------------+---------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined
+(1 row)
+
+\dX+
+                                                                   List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+----------+------------------------+--------------------------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        |         |                | 106 bytes         | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built   |                |                   | 24 kB
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built   |                |                   | 386 bytes
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built   |                |                   | 294 bytes
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              |         | 13 bytes       |                   | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined |                | 0 bytes           | 0 bytes
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built   |                |                   | 686 bytes
+(12 rows)
+
+\dX+ stts_?
+                                                List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+--------+--------+-------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public | stts_1 | a, b FROM stts_t1 | built     |              |         | 13 bytes       |                   | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(4 rows)
+
+\dX+ *stts_hoge
+                                                       List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+--------+-----------+-------------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(1 row)
+
+\dX+ stts_s2.stts_yama
+                                                    List of extended statistics
+ Schema  |   Name    |       Definition        | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+---------+-----------+-------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ stts_s2 | stts_yama | col1, col3 FROM stts_t3 |           | defined      | defined |                | 0 bytes           | 0 bytes
+(1 row)
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 7912e733ae..db6e3e1ba3 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -912,6 +912,37 @@ CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
 
 ANALYZE tststats.priv_test_tbl;
 
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+
+\dX
+\dX stts_?
+\dX *stts_hoge
+\dX+
+\dX+ stts_?
+\dX+ *stts_hoge
+\dX+ stts_s2.stts_yama
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
+
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
-- 
2.26.2


--------------B0278A9E952F6668D2DBEBA1
Content-Type: text/x-patch; charset=UTF-8;
 name="0002-fixup-rename-defined-to-requested-20210109.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="0002-fixup-rename-defined-to-requested-20210109.patch"



^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* [PATCH 1/2] psql \dX: list extended statistics objects
@ 2021-01-07 05:28 Tatsuro Yamada <[email protected]>
  0 siblings, 0 replies; 30+ messages in thread

From: Tatsuro Yamada @ 2021-01-07 05:28 UTC (permalink / raw)

The new command lists extended statistics objects, possibly with their
sizes. All past releases with extended statistics are supported.

Author: Tatsuro Yamada
Reviewed-by: Julien Rouhaud, Alvaro Herrera, Tomas Vondra
Discussion: https://postgr.es/m/c027a541-5856-75a5-0868-341301e1624b%40nttcom.co.jp_1
---
 doc/src/sgml/ref/psql-ref.sgml          |  14 +++
 src/bin/psql/command.c                  |   3 +
 src/bin/psql/describe.c                 | 150 ++++++++++++++++++++++++
 src/bin/psql/describe.h                 |   3 +
 src/bin/psql/help.c                     |   1 +
 src/bin/psql/tab-complete.c             |   4 +-
 src/test/regress/expected/stats_ext.out |  94 +++++++++++++++
 src/test/regress/sql/stats_ext.sql      |  31 +++++
 8 files changed, 299 insertions(+), 1 deletion(-)

diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml
index 221a967bfe..d01acc92b8 100644
--- a/doc/src/sgml/ref/psql-ref.sgml
+++ b/doc/src/sgml/ref/psql-ref.sgml
@@ -1918,6 +1918,20 @@ testdb=&gt;
         </para>
         </listitem>
       </varlistentry>
+      
+      <varlistentry>
+        <term><literal>\dX [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
+        <listitem>
+        <para>
+        Lists extended statistics.
+        If <replaceable class="parameter">pattern</replaceable>
+        is specified, only those extended statistics whose names match the
+        pattern are listed.
+        If <literal>+</literal> is appended to the command name, each extended
+        statistics is listed with its size.
+        </para>
+        </listitem>
+      </varlistentry>
 
       <varlistentry>
         <term><literal>\dy[+] [ <link linkend="app-psql-patterns"><replaceable class="parameter">pattern</replaceable></link> ]</literal></term>
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index 303e7c3ad8..c5ebc1c3f4 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -928,6 +928,9 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd)
 				else
 					success = listExtensions(pattern);
 				break;
+			case 'X':			/* Extended Statistics */
+				success = listExtendedStats(pattern, show_verbose);
+				break;
 			case 'y':			/* Event Triggers */
 				success = listEventTriggers(pattern, show_verbose);
 				break;
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index caf97563f4..46f54199fb 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -4392,6 +4392,156 @@ listEventTriggers(const char *pattern, bool verbose)
 	return true;
 }
 
+/*
+ * \dX
+ *
+ * Describes extended statistics.
+ */
+bool
+listExtendedStats(const char *pattern, bool verbose)
+{
+	PQExpBufferData buf;
+	PGresult   *res;
+	printQueryOpt myopt = pset.popt;
+
+	if (pset.sversion < 100000)
+	{
+		char		sverbuf[32];
+
+		pg_log_error("The server (version %s) does not support extended statistics.",
+					 formatPGVersionNumber(pset.sversion, false,
+										   sverbuf, sizeof(sverbuf)));
+		return true;
+	}
+
+	initPQExpBuffer(&buf);
+	printfPQExpBuffer(&buf,
+					  "SELECT \n"
+					  "es.stxnamespace::pg_catalog.regnamespace::text AS \"%s\", \n"
+					  "es.stxname AS \"%s\", \n"
+					  "pg_catalog.format('%%s FROM %%s', \n"
+					  "  (SELECT pg_catalog.string_agg(pg_catalog.quote_ident(a.attname),', ') \n"
+					  "   FROM pg_catalog.unnest(es.stxkeys) s(attnum) \n"
+					  "   JOIN pg_catalog.pg_attribute a \n"
+					  "   ON (es.stxrelid = a.attrelid \n"
+					  "   AND a.attnum = s.attnum \n"
+					  "   AND NOT a.attisdropped)), \n"
+					  "es.stxrelid::regclass) AS \"%s\"",
+					  gettext_noop("Schema"),
+					  gettext_noop("Name"),
+					  gettext_noop("Definition"));
+
+	/*
+	 * Since 12 there are two catalogs - one for the definition, one for the
+	 * data built by ANALYZE. Older releases use a single catalog. Also, 12
+	 * adds the MCV statistics kind.
+	 */
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN es.stxdependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"));
+	}
+	else
+	{
+		appendPQExpBuffer(&buf,
+						  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'd' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxddependencies IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'f' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\", \n"
+						  "CASE WHEN esd.stxdmcv IS NOT NULL THEN 'built' \n"
+						  "     WHEN 'm' = any(es.stxkind) THEN 'defined' \n"
+						  "END AS \"%s\"",
+						  gettext_noop("Ndistinct"),
+						  gettext_noop("Dependencies"),
+						  gettext_noop("MCV"));
+	}
+
+	/* In verbose mode, print sizes of the extended statistics objects. */
+	if (verbose)
+	{
+		if (pset.sversion < 120000)
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN es.stxndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN es.stxdependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(es.stxdependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"));
+		}
+		else
+		{
+			appendPQExpBuffer(&buf,
+							  ",\nCASE WHEN esd.stxdndistinct IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdndistinct)::bigint) \n"
+							  "     WHEN 'd' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxddependencies IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxddependencies)::bigint) \n"
+							  "     WHEN 'f' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\", \n"
+							  "CASE WHEN esd.stxdmcv IS NOT NULL THEN \n"
+							  "          pg_catalog.pg_size_pretty(pg_catalog.length(esd.stxdmcv)::bigint) \n"
+							  "     WHEN 'm' = any(es.stxkind) THEN '0 bytes' \n"
+							  "END AS \"%s\"",
+							  gettext_noop("Ndistinct_size"),
+							  gettext_noop("Dependencies_size"),
+							  gettext_noop("MCV_size"));
+		}
+	}
+
+	if (pset.sversion < 120000)
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+	else
+	{
+		appendPQExpBufferStr(&buf,
+							 " \nFROM pg_catalog.pg_statistic_ext es \n"
+							 "LEFT JOIN pg_catalog.pg_statistic_ext_data esd \n"
+							 "ON es.oid = esd.stxoid \n"
+							 "INNER JOIN pg_catalog.pg_class c \n"
+							 "ON es.stxrelid = c.oid \n");
+	}
+
+	processSQLNamePattern(pset.db, &buf, pattern, false,
+						  false, NULL,
+						  "stxname", NULL,
+						  NULL);
+
+	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
+
+	res = PSQLexec(buf.data);
+	termPQExpBuffer(&buf);
+	if (!res)
+		return false;
+
+	myopt.nullPrint = NULL;
+	myopt.title = _("List of extended statistics");
+	myopt.translate_header = true;
+
+	printQuery(res, &myopt, pset.queryFout, false, pset.logfile);
+
+	PQclear(res);
+	return true;
+}
+
 /*
  * \dC
  *
diff --git a/src/bin/psql/describe.h b/src/bin/psql/describe.h
index 6044e3a082..867e57d851 100644
--- a/src/bin/psql/describe.h
+++ b/src/bin/psql/describe.h
@@ -102,6 +102,9 @@ extern bool listExtensions(const char *pattern);
 /* \dx+ */
 extern bool listExtensionContents(const char *pattern);
 
+/* \dX */
+extern bool listExtendedStats(const char *pattern, bool verbose);
+
 /* \dy */
 extern bool listEventTriggers(const char *pattern, bool verbose);
 
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 9ec1c4e810..e42bc8c54e 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -267,6 +267,7 @@ slashUsage(unsigned short int pager)
 	fprintf(output, _("  \\du[S+] [PATTERN]      list roles\n"));
 	fprintf(output, _("  \\dv[S+] [PATTERN]      list views\n"));
 	fprintf(output, _("  \\dx[+]  [PATTERN]      list extensions\n"));
+	fprintf(output, _("  \\dX[+]  [PATTERN]      list extended statistics\n"));
 	fprintf(output, _("  \\dy     [PATTERN]      list event triggers\n"));
 	fprintf(output, _("  \\l[+]   [PATTERN]      list databases\n"));
 	fprintf(output, _("  \\sf[+]  FUNCNAME       show a function's definition\n"));
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9dcab0d2fa..611f1efb15 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1500,7 +1500,7 @@ psql_completion(const char *text, int start, int end)
 		"\\dF", "\\dFd", "\\dFp", "\\dFt", "\\dg", "\\di", "\\dl", "\\dL",
 		"\\dm", "\\dn", "\\do", "\\dO", "\\dp", "\\dP", "\\dPi", "\\dPt",
 		"\\drds", "\\dRs", "\\dRp", "\\ds", "\\dS",
-		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dy",
+		"\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dX", "\\dy",
 		"\\e", "\\echo", "\\ef", "\\elif", "\\else", "\\encoding",
 		"\\endif", "\\errverbose", "\\ev",
 		"\\f",
@@ -3910,6 +3910,8 @@ psql_completion(const char *text, int start, int end)
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_views, NULL);
 	else if (TailMatchesCS("\\dx*"))
 		COMPLETE_WITH_QUERY(Query_for_list_of_extensions);
+	else if (TailMatchesCS("\\dX*"))
+		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_statistics, NULL);
 	else if (TailMatchesCS("\\dm*"))
 		COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_matviews, NULL);
 	else if (TailMatchesCS("\\dE*"))
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 7bfeaf85f0..8c8a0afcf6 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -1725,6 +1725,100 @@ INSERT INTO tststats.priv_test_tbl
 CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
   FROM tststats.priv_test_tbl;
 ANALYZE tststats.priv_test_tbl;
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+\dX
+                                          List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   
+----------+------------------------+--------------------------------------+-----------+--------------+---------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built
+(12 rows)
+
+\dX stts_?
+                       List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   
+--------+--------+-------------------+-----------+--------------+---------
+ public | stts_1 | a, b FROM stts_t1 | built     |              | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined
+(4 rows)
+
+\dX *stts_hoge
+                               List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   
+--------+-----------+-------------------------------+-----------+--------------+---------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined
+(1 row)
+
+\dX+
+                                                                   List of extended statistics
+  Schema  |          Name          |              Definition              | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+----------+------------------------+--------------------------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public   | func_deps_stat         | a, b, c FROM functional_dependencies |           | built        |         |                | 106 bytes         | 
+ public   | mcv_lists_arrays_stats | a, b, c FROM mcv_lists_arrays        |           |              | built   |                |                   | 24 kB
+ public   | mcv_lists_bool_stats   | a, b, c FROM mcv_lists_bool          |           |              | built   |                |                   | 386 bytes
+ public   | mcv_lists_stats        | a, b, d FROM mcv_lists               |           |              | built   |                |                   | 294 bytes
+ public   | stts_1                 | a, b FROM stts_t1                    | built     |              |         | 13 bytes       |                   | 
+ public   | stts_2                 | a, b FROM stts_t1                    | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public   | stts_3                 | a, b FROM stts_t1                    | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public   | stts_4                 | b, c FROM stts_t2                    | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ public   | stts_hoge              | col1, col2, col3 FROM stts_t3        | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s1  | stts_foo               | col1, col2 FROM stts_t3              | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+ stts_s2  | stts_yama              | col1, col3 FROM stts_t3              |           | defined      | defined |                | 0 bytes           | 0 bytes
+ tststats | priv_test_stats        | a, b FROM tststats.priv_test_tbl     |           |              | built   |                |                   | 686 bytes
+(12 rows)
+
+\dX+ stts_?
+                                                List of extended statistics
+ Schema |  Name  |    Definition     | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size |  MCV_size  
+--------+--------+-------------------+-----------+--------------+---------+----------------+-------------------+------------
+ public | stts_1 | a, b FROM stts_t1 | built     |              |         | 13 bytes       |                   | 
+ public | stts_2 | a, b FROM stts_t1 | built     | built        |         | 13 bytes       | 40 bytes          | 
+ public | stts_3 | a, b FROM stts_t1 | built     | built        | built   | 13 bytes       | 40 bytes          | 6126 bytes
+ public | stts_4 | b, c FROM stts_t2 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(4 rows)
+
+\dX+ *stts_hoge
+                                                       List of extended statistics
+ Schema |   Name    |          Definition           | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+--------+-----------+-------------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ public | stts_hoge | col1, col2, col3 FROM stts_t3 | defined   | defined      | defined | 0 bytes        | 0 bytes           | 0 bytes
+(1 row)
+
+\dX+ stts_s2.stts_yama
+                                                    List of extended statistics
+ Schema  |   Name    |       Definition        | Ndistinct | Dependencies |   MCV   | Ndistinct_size | Dependencies_size | MCV_size 
+---------+-----------+-------------------------+-----------+--------------+---------+----------------+-------------------+----------
+ stts_s2 | stts_yama | col1, col3 FROM stts_t3 |           | defined      | defined |                | 0 bytes           | 0 bytes
+(1 row)
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 7912e733ae..db6e3e1ba3 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -912,6 +912,37 @@ CREATE STATISTICS tststats.priv_test_stats (mcv) ON a, b
 
 ANALYZE tststats.priv_test_tbl;
 
+-- Check printing info about extended statistics by \dX
+create table stts_t1 (a int, b int);
+create statistics stts_1 (ndistinct) on a, b from stts_t1;
+create statistics stts_2 (ndistinct, dependencies) on a, b from stts_t1;
+create statistics stts_3 (ndistinct, dependencies, mcv) on a, b from stts_t1;
+
+create table stts_t2 (a int, b int, c int);
+create statistics stts_4 on b, c from stts_t2;
+
+create table stts_t3 (col1 int, col2 int, col3 int);
+create statistics stts_hoge on col1, col2, col3 from stts_t3;
+
+create schema stts_s1;
+create schema stts_s2;
+create statistics stts_s1.stts_foo on col1, col2 from stts_t3;
+create statistics stts_s2.stts_yama (dependencies, mcv) on col1, col3 from stts_t3;
+
+insert into stts_t1 select i,i from generate_series(1,100) i;
+analyze stts_t1;
+
+\dX
+\dX stts_?
+\dX *stts_hoge
+\dX+
+\dX+ stts_?
+\dX+ *stts_hoge
+\dX+ stts_s2.stts_yama
+
+drop table stts_t1, stts_t2, stts_t3;
+drop schema stts_s1, stts_s2 cascade;
+
 -- User with no access
 CREATE USER regress_stats_user1;
 GRANT USAGE ON SCHEMA tststats TO regress_stats_user1;
-- 
2.26.2


--------------B0278A9E952F6668D2DBEBA1
Content-Type: text/x-patch; charset=UTF-8;
 name="0002-fixup-rename-defined-to-requested-20210109.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="0002-fixup-rename-defined-to-requested-20210109.patch"



^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* Re: ci: Allow running mingw tests by default via environment variable
@ 2024-04-25 12:02 Nazir Bilal Yavuz <[email protected]>
  2025-03-04 00:27 ` Re: ci: Allow running mingw tests by default via environment variable Thomas Munro <[email protected]>
  0 siblings, 1 reply; 30+ messages in thread

From: Nazir Bilal Yavuz @ 2024-04-25 12:02 UTC (permalink / raw)
  To: Andres Freund <[email protected]>; +Cc: pgsql-hackers; Wolfgang Walther <[email protected]>; Thomas Munro <[email protected]>

Hi,

Thank you for working on this!

On Sat, 13 Apr 2024 at 05:12, Andres Freund <[email protected]> wrote:
>
> Hi,
>
> We have CI support for mingw, but don't run the task by default, as it eats up
> precious CI credits.  However, for cfbot we are using custom compute resources
> (currently donated by google), so we can afford to run the mingw tests. Right
> now that'd require cfbot to patch .cirrus.tasks.yml.

And I think mingw ends up not running most of the time. +1 to running
it as default at least on cfbot. Also, this gives people a chance to
run mingw as default on their personal repositories (which I would
like to run).

> While one can manually trigger manual task in one one's own repo, most won't
> have the permissions to do so for cfbot.
>
>
> I propose that we instead run the task automatically if
> $REPO_MINGW_TRIGGER_BY_DEFAULT is set, typically in cirrus' per-repository
> configuration.
>
> Unfortunately that's somewhat awkward to do in the cirrus-ci yaml
> configuration, the set of conditional expressions supported is very
> simplistic.
>
> To deal with that, I extended .cirrus.star to compute the required environment
> variable. If $REPO_MINGW_TRIGGER_BY_DEFAULT is set, CI_MINGW_TRIGGER_TYPE is
> set to 'automatic', if not it's 'manual'.
>

Changes look good to me. My only complaint could be using only 'true'
for the REPO_MINGW_TRIGGER_BY_DEFAULT, not a possible list of values
but this is not important.

> We've also talked in other threads about adding CI support for
> 1) windows, building with visual studio
> 2) linux, with musl libc
> 3) free/netbsd
>
> That becomes more enticing, if we can enable them by default on cfbot but not
> elsewhere. With this change, it'd be easy to add further variables to control
> such future tasks.

I agree.

> I also attached a second commit, that makes the "code" dealing with ci-os-only
> in .cirrus.tasks.yml simpler. While I think it does nicely simplify
> .cirrus.tasks.yml, overall it adds lines, possibly making this not worth it.
> I'm somewhat on the fence.
>
>
> Thoughts?

Although it adds more lines, this makes the .cirrus.tasks.yml file
more readable and understandable which is more important in my
opinion.

> On the code level, I thought if it'd be good to have a common prefix for all
> the automatically set variables. Right now that's CI_, but I'm not at all
> wedded to that.

I agree with your thoughts and CI_ prefix.

I tested both patches and they work as expected.

-- 
Regards,
Nazir Bilal Yavuz
Microsoft






^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* Re: ci: Allow running mingw tests by default via environment variable
  2024-04-25 12:02 Re: ci: Allow running mingw tests by default via environment variable Nazir Bilal Yavuz <[email protected]>
@ 2025-03-04 00:27 ` Thomas Munro <[email protected]>
  2025-03-04 08:53   ` Re: ci: Allow running mingw tests by default via environment variable Nazir Bilal Yavuz <[email protected]>
  2025-03-04 12:58   ` Re: ci: Allow running mingw tests by default via environment variable Andres Freund <[email protected]>
  0 siblings, 2 replies; 30+ messages in thread

From: Thomas Munro @ 2025-03-04 00:27 UTC (permalink / raw)
  To: Nazir Bilal Yavuz <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers; Wolfgang Walther <[email protected]>

On Fri, Apr 26, 2024 at 12:02 AM Nazir Bilal Yavuz <[email protected]> wrote:
> On Sat, 13 Apr 2024 at 05:12, Andres Freund <[email protected]> wrote:
> > I propose that we instead run the task automatically if
> > $REPO_MINGW_TRIGGER_BY_DEFAULT is set, typically in cirrus' per-repository
> > configuration.
> >
> > Unfortunately that's somewhat awkward to do in the cirrus-ci yaml
> > configuration, the set of conditional expressions supported is very
> > simplistic.
> >
> > To deal with that, I extended .cirrus.star to compute the required environment
> > variable. If $REPO_MINGW_TRIGGER_BY_DEFAULT is set, CI_MINGW_TRIGGER_TYPE is
> > set to 'automatic', if not it's 'manual'.
>
> Changes look good to me. My only complaint could be using only 'true'
> for the REPO_MINGW_TRIGGER_BY_DEFAULT, not a possible list of values
> but this is not important.

Here's a generalised version of 0001.  If you put this in your
repository settings on Cirrus's website:

REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw netbsd"

... then it defines:

CI_TRIGGER_TYPE_MINGW=automatic
CI_TRIGGER_TYPE_NETBSD=automatic
CI_TRIGGER_TYPE_OPENBSD=manual

The set of tasks that get this treatment is defined by this line in
.cirrus.star:

    default_manual_trigger_tasks = ['mingw', 'netbsd', 'openbsd']

Then the individual tasks in .cirrus.tasks.yml use those variables:

     - name: NetBSD - Meson
+      # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
+      trigger_type: $CI_TRIGGER_TYPE_NETBSD

Unfortunately the funky Starlark language doesn't seem to have regular
expressions, so it's just searching for those substrings without
contemplating delimiters.  That doesn't seem to be a problem at this
scale...

(I didn't look at the second patch today.)


Attachments:

  [text/x-patch] v3-0001-ci-Per-repo-triggers-for-MinGW-NetBSD-OpenBSD.patch (7.2K, ../../CA+hUKGKeU=YStaX_3AjJakTBOO_dBTPj4mf8sEkQOVjLqs7qEA@mail.gmail.com/2-v3-0001-ci-Per-repo-triggers-for-MinGW-NetBSD-OpenBSD.patch)
  download | inline diff:
From 74f4fb7a5af0b53eb4caf7de0ee186ffaaf6ee20 Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Fri, 12 Apr 2024 15:04:32 -0700
Subject: [PATCH v3] ci: Per-repo triggers for MinGW, NetBSD, OpenBSD.

We're not ready to trigger these tasks automatically by default, so you
normally have to trigger then manually via Cirrus if you want to run
them.  This avoids using too many credits by default.

With this commit, an individual repository can be configured to trigger
them automatically using an environment variable defined under
"Repository Settings", for example:

REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw netbsd openbsd"

This will enable cfbot to turn them on by default when running tests for
the Commitfest app.  It is not subject to free credit limits as it runs
on credits donated by Google.

Author: Andres Freund <[email protected]>
Co-authored-by: Thomas Munro <[email protected]>
Reviewed-by: Nazir Bilal Yavuz <[email protected]>
Discussion: https://postgr.es/m/20240413021221.hg53rvqlvldqh57i%40awork3.anarazel.de
---
 .cirrus.star        | 50 +++++++++++++++++++++++++++++++++++++++++----
 .cirrus.tasks.yml   | 15 +++++++-------
 .cirrus.yml         | 12 +++++++++--
 src/tools/ci/README | 11 ++++++++++
 4 files changed, 75 insertions(+), 13 deletions(-)

diff --git a/.cirrus.star b/.cirrus.star
index 36233872d1e..218fce887b5 100644
--- a/.cirrus.star
+++ b/.cirrus.star
@@ -7,7 +7,7 @@ https://github.com/bazelbuild/starlark/blob/master/spec.md
 See also .cirrus.yml and src/tools/ci/README
 """
 
-load("cirrus", "env", "fs")
+load("cirrus", "env", "fs", "yaml")
 
 
 def main():
@@ -18,19 +18,36 @@ def main():
 
     1) the contents of .cirrus.yml
 
-    2) if defined, the contents of the file referenced by the, repository
+    2) computed environment variables
+
+    3) if defined, the contents of the file referenced by the, repository
        level, REPO_CI_CONFIG_GIT_URL variable (see
        https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
        format)
 
-    3) .cirrus.tasks.yml
+    4) .cirrus.tasks.yml
     """
 
     output = ""
 
     # 1) is evaluated implicitly
 
+
     # Add 2)
+    additional_env = compute_environment_vars()
+    env_fmt = """
+###
+# Computed environment variables start here
+###
+{0}
+###
+# Computed environment variables end here
+###
+"""
+    output += env_fmt.format(yaml.dumps({'env': additional_env}))
+
+
+    # Add 3)
     repo_config_url = env.get("REPO_CI_CONFIG_GIT_URL")
     if repo_config_url != None:
         print("loading additional configuration from \"{}\"".format(repo_config_url))
@@ -38,12 +55,37 @@ def main():
     else:
         output += "\n# REPO_CI_CONFIG_URL was not set\n"
 
-    # Add 3)
+
+    # Add 4)
     output += config_from(".cirrus.tasks.yml")
 
+
     return output
 
 
+def compute_environment_vars():
+    cenv = {}
+
+    # The following tasks are manually triggered by default because they
+    # might use too many resources for users of free Cirrus credits, but you can
+    # trigger them automatically by naming them in an environment variable eg
+    # REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw netbsd" under "Repository
+    # Settings" on Cirrus CI's website.
+
+    default_manual_trigger_tasks = ['mingw', 'netbsd', 'openbsd']
+
+    repo_ci_automatic_trigger_tasks = env.get('REPO_CI_AUTOMATIC_TRIGGER_TASKS', '')
+    for task in default_manual_trigger_tasks:
+        name = 'CI_TRIGGER_TYPE_' + task.upper()
+        if repo_ci_automatic_trigger_tasks.find(task) != -1:
+            value = 'automatic'
+        else:
+            value = 'manual'
+        cenv[name] = value
+
+    return cenv
+
+
 def config_from(config_src):
     """return contents of config file `config_src`, surrounded by markers
     indicating start / end of the included file
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index c5e7b743bfb..f5ff0ccf222 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -218,7 +218,6 @@ task:
 
 task:
   depends_on: SanityCheck
-  trigger_type: manual
 
   env:
     # Below are experimentally derived to be a decent choice.
@@ -236,6 +235,8 @@ task:
 
   matrix:
     - name: NetBSD - Meson
+      # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
+      trigger_type: $CI_TRIGGER_TYPE_NETBSD
       only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*netbsd.*'
       env:
         OS_NAME: netbsd
@@ -253,6 +254,8 @@ task:
       <<: *netbsd_task_template
 
     - name: OpenBSD - Meson
+      # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
+      trigger_type: $CI_TRIGGER_TYPE_OPENBSD
       only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*openbsd.*'
       env:
         OS_NAME: openbsd
@@ -706,13 +709,11 @@ task:
   << : *WINDOWS_ENVIRONMENT_BASE
   name: Windows - Server 2019, MinGW64 - Meson
 
-  # due to resource constraints we don't run this task by default for now
-  trigger_type: manual
-  # worth using only_if despite being manual, otherwise this task will show up
-  # when e.g. ci-os-only: linux is used.
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
-  # otherwise it'll be sorted before other tasks
+  # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star.
+  trigger_type: $CI_TRIGGER_TYPE_MINGW
+
   depends_on: SanityCheck
+  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
 
   env:
     TEST_JOBS: 4 # higher concurrency causes occasional failures
diff --git a/.cirrus.yml b/.cirrus.yml
index 33c6e481d74..3f75852e84e 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -10,12 +10,20 @@
 #
 # 1) the contents of this file
 #
-# 2) if defined, the contents of the file referenced by the, repository
+# 2) computed environment variables
+#
+#    Used to enable/disable tasks based on the execution environment. See
+#    .cirrus.star: compute_environment_vars()
+#
+# 3) if defined, the contents of the file referenced by the, repository
 #    level, REPO_CI_CONFIG_GIT_URL variable (see
 #    https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
 #    format)
 #
-# 3) .cirrus.tasks.yml
+#    This allows running tasks in a different execution environment than the
+#    default, e.g. to have sufficient resources for cfbot.
+#
+# 4) .cirrus.tasks.yml
 #
 # This composition is done by .cirrus.star
 
diff --git a/src/tools/ci/README b/src/tools/ci/README
index 12c1e7c308f..d183648a8d0 100644
--- a/src/tools/ci/README
+++ b/src/tools/ci/README
@@ -82,3 +82,14 @@ defined in .cirrus.yml, by redefining the relevant yaml anchors.
 Custom compute resources can be provided using
 - https://cirrus-ci.org/guide/supported-computing-services/
 - https://cirrus-ci.org/guide/persistent-workers/
+
+
+Enabling manual tasks by default
+================================
+
+Some tasks are not triggered automatically by default, to avoid using up CI
+credits too quickly. This can be changed on the repository level, e.g. when
+custom compute resources are configured.
+
+The following repository level environment variables are recognized:
+- REPO_CI_AUTOMATIC_TRIGGER_TASKS - space-separated list of (mingw|netbsd|openbsd)
-- 
2.48.1



^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* Re: ci: Allow running mingw tests by default via environment variable
  2024-04-25 12:02 Re: ci: Allow running mingw tests by default via environment variable Nazir Bilal Yavuz <[email protected]>
  2025-03-04 00:27 ` Re: ci: Allow running mingw tests by default via environment variable Thomas Munro <[email protected]>
@ 2025-03-04 08:53   ` Nazir Bilal Yavuz <[email protected]>
  2025-03-04 13:00     ` Re: ci: Allow running mingw tests by default via environment variable Andres Freund <[email protected]>
  1 sibling, 1 reply; 30+ messages in thread

From: Nazir Bilal Yavuz @ 2025-03-04 08:53 UTC (permalink / raw)
  To: Thomas Munro <[email protected]>; +Cc: Andres Freund <[email protected]>; pgsql-hackers; Wolfgang Walther <[email protected]>

Hi,

On Tue, 4 Mar 2025 at 03:28, Thomas Munro <[email protected]> wrote:
>
> On Fri, Apr 26, 2024 at 12:02 AM Nazir Bilal Yavuz <[email protected]> wrote:
> > On Sat, 13 Apr 2024 at 05:12, Andres Freund <[email protected]> wrote:
> > > I propose that we instead run the task automatically if
> > > $REPO_MINGW_TRIGGER_BY_DEFAULT is set, typically in cirrus' per-repository
> > > configuration.
> > >
> > > Unfortunately that's somewhat awkward to do in the cirrus-ci yaml
> > > configuration, the set of conditional expressions supported is very
> > > simplistic.
> > >
> > > To deal with that, I extended .cirrus.star to compute the required environment
> > > variable. If $REPO_MINGW_TRIGGER_BY_DEFAULT is set, CI_MINGW_TRIGGER_TYPE is
> > > set to 'automatic', if not it's 'manual'.
> >
> > Changes look good to me. My only complaint could be using only 'true'
> > for the REPO_MINGW_TRIGGER_BY_DEFAULT, not a possible list of values
> > but this is not important.
>
> Here's a generalised version of 0001.  If you put this in your
> repository settings on Cirrus's website:

Thank you for working on this!

> REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw netbsd"
>
> ... then it defines:
>
> CI_TRIGGER_TYPE_MINGW=automatic
> CI_TRIGGER_TYPE_NETBSD=automatic
> CI_TRIGGER_TYPE_OPENBSD=manual
>
> The set of tasks that get this treatment is defined by this line in
> .cirrus.star:
>
>     default_manual_trigger_tasks = ['mingw', 'netbsd', 'openbsd']
>
> Then the individual tasks in .cirrus.tasks.yml use those variables:
>
>      - name: NetBSD - Meson
> +      # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
> +      trigger_type: $CI_TRIGGER_TYPE_NETBSD

I confirm that this works as expected and LGTM.

> Unfortunately the funky Starlark language doesn't seem to have regular
> expressions, so it's just searching for those substrings without
> contemplating delimiters.  That doesn't seem to be a problem at this
> scale...

You can load 're' and use it as in the v2-0002 but I think what you
did is good enough.

I rebased Andres' v2-0002 on top of recent changes and wrote a small
commit message. v4 is attached.

--
Regards,
Nazir Bilal Yavuz
Microsoft


Attachments:

  [text/x-patch] v4-0001-ci-Per-repo-triggers-for-MinGW-NetBSD-OpenBSD.patch (7.2K, ../../CAN55FZ1cY7iFj98F=8WyKvvwesGTVoKs5n95y9GqgNRNotcYbA@mail.gmail.com/2-v4-0001-ci-Per-repo-triggers-for-MinGW-NetBSD-OpenBSD.patch)
  download | inline diff:
From 566a729bf13a5e5c782ea367780cce68ae839b92 Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Fri, 12 Apr 2024 15:04:32 -0700
Subject: [PATCH v4 1/2] ci: Per-repo triggers for MinGW, NetBSD, OpenBSD.

We're not ready to trigger these tasks automatically by default, so you
normally have to trigger then manually via Cirrus if you want to run
them.  This avoids using too many credits by default.

With this commit, an individual repository can be configured to trigger
them automatically using an environment variable defined under
"Repository Settings", for example:

REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw netbsd openbsd"

This will enable cfbot to turn them on by default when running tests for
the Commitfest app.  It is not subject to free credit limits as it runs
on credits donated by Google.

Author: Andres Freund <[email protected]>
Co-authored-by: Thomas Munro <[email protected]>
Reviewed-by: Nazir Bilal Yavuz <[email protected]>
Discussion: https://postgr.es/m/20240413021221.hg53rvqlvldqh57i%40awork3.anarazel.de
---
 .cirrus.star        | 50 +++++++++++++++++++++++++++++++++++++++++----
 .cirrus.tasks.yml   | 15 +++++++-------
 .cirrus.yml         | 12 +++++++++--
 src/tools/ci/README | 11 ++++++++++
 4 files changed, 75 insertions(+), 13 deletions(-)

diff --git a/.cirrus.star b/.cirrus.star
index 36233872d1e..218fce887b5 100644
--- a/.cirrus.star
+++ b/.cirrus.star
@@ -7,7 +7,7 @@ https://github.com/bazelbuild/starlark/blob/master/spec.md
 See also .cirrus.yml and src/tools/ci/README
 """
 
-load("cirrus", "env", "fs")
+load("cirrus", "env", "fs", "yaml")
 
 
 def main():
@@ -18,19 +18,36 @@ def main():
 
     1) the contents of .cirrus.yml
 
-    2) if defined, the contents of the file referenced by the, repository
+    2) computed environment variables
+
+    3) if defined, the contents of the file referenced by the, repository
        level, REPO_CI_CONFIG_GIT_URL variable (see
        https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
        format)
 
-    3) .cirrus.tasks.yml
+    4) .cirrus.tasks.yml
     """
 
     output = ""
 
     # 1) is evaluated implicitly
 
+
     # Add 2)
+    additional_env = compute_environment_vars()
+    env_fmt = """
+###
+# Computed environment variables start here
+###
+{0}
+###
+# Computed environment variables end here
+###
+"""
+    output += env_fmt.format(yaml.dumps({'env': additional_env}))
+
+
+    # Add 3)
     repo_config_url = env.get("REPO_CI_CONFIG_GIT_URL")
     if repo_config_url != None:
         print("loading additional configuration from \"{}\"".format(repo_config_url))
@@ -38,12 +55,37 @@ def main():
     else:
         output += "\n# REPO_CI_CONFIG_URL was not set\n"
 
-    # Add 3)
+
+    # Add 4)
     output += config_from(".cirrus.tasks.yml")
 
+
     return output
 
 
+def compute_environment_vars():
+    cenv = {}
+
+    # The following tasks are manually triggered by default because they
+    # might use too many resources for users of free Cirrus credits, but you can
+    # trigger them automatically by naming them in an environment variable eg
+    # REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw netbsd" under "Repository
+    # Settings" on Cirrus CI's website.
+
+    default_manual_trigger_tasks = ['mingw', 'netbsd', 'openbsd']
+
+    repo_ci_automatic_trigger_tasks = env.get('REPO_CI_AUTOMATIC_TRIGGER_TASKS', '')
+    for task in default_manual_trigger_tasks:
+        name = 'CI_TRIGGER_TYPE_' + task.upper()
+        if repo_ci_automatic_trigger_tasks.find(task) != -1:
+            value = 'automatic'
+        else:
+            value = 'manual'
+        cenv[name] = value
+
+    return cenv
+
+
 def config_from(config_src):
     """return contents of config file `config_src`, surrounded by markers
     indicating start / end of the included file
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index c5e7b743bfb..f5ff0ccf222 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -218,7 +218,6 @@ task:
 
 task:
   depends_on: SanityCheck
-  trigger_type: manual
 
   env:
     # Below are experimentally derived to be a decent choice.
@@ -236,6 +235,8 @@ task:
 
   matrix:
     - name: NetBSD - Meson
+      # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
+      trigger_type: $CI_TRIGGER_TYPE_NETBSD
       only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*netbsd.*'
       env:
         OS_NAME: netbsd
@@ -253,6 +254,8 @@ task:
       <<: *netbsd_task_template
 
     - name: OpenBSD - Meson
+      # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
+      trigger_type: $CI_TRIGGER_TYPE_OPENBSD
       only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*openbsd.*'
       env:
         OS_NAME: openbsd
@@ -706,13 +709,11 @@ task:
   << : *WINDOWS_ENVIRONMENT_BASE
   name: Windows - Server 2019, MinGW64 - Meson
 
-  # due to resource constraints we don't run this task by default for now
-  trigger_type: manual
-  # worth using only_if despite being manual, otherwise this task will show up
-  # when e.g. ci-os-only: linux is used.
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
-  # otherwise it'll be sorted before other tasks
+  # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star.
+  trigger_type: $CI_TRIGGER_TYPE_MINGW
+
   depends_on: SanityCheck
+  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
 
   env:
     TEST_JOBS: 4 # higher concurrency causes occasional failures
diff --git a/.cirrus.yml b/.cirrus.yml
index 33c6e481d74..3f75852e84e 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -10,12 +10,20 @@
 #
 # 1) the contents of this file
 #
-# 2) if defined, the contents of the file referenced by the, repository
+# 2) computed environment variables
+#
+#    Used to enable/disable tasks based on the execution environment. See
+#    .cirrus.star: compute_environment_vars()
+#
+# 3) if defined, the contents of the file referenced by the, repository
 #    level, REPO_CI_CONFIG_GIT_URL variable (see
 #    https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
 #    format)
 #
-# 3) .cirrus.tasks.yml
+#    This allows running tasks in a different execution environment than the
+#    default, e.g. to have sufficient resources for cfbot.
+#
+# 4) .cirrus.tasks.yml
 #
 # This composition is done by .cirrus.star
 
diff --git a/src/tools/ci/README b/src/tools/ci/README
index 12c1e7c308f..d183648a8d0 100644
--- a/src/tools/ci/README
+++ b/src/tools/ci/README
@@ -82,3 +82,14 @@ defined in .cirrus.yml, by redefining the relevant yaml anchors.
 Custom compute resources can be provided using
 - https://cirrus-ci.org/guide/supported-computing-services/
 - https://cirrus-ci.org/guide/persistent-workers/
+
+
+Enabling manual tasks by default
+================================
+
+Some tasks are not triggered automatically by default, to avoid using up CI
+credits too quickly. This can be changed on the repository level, e.g. when
+custom compute resources are configured.
+
+The following repository level environment variables are recognized:
+- REPO_CI_AUTOMATIC_TRIGGER_TASKS - space-separated list of (mingw|netbsd|openbsd)
-- 
2.47.2



  [text/x-patch] v4-0002-ci-Simplify-ci-os-only-handling.patch (5.8K, ../../CAN55FZ1cY7iFj98F=8WyKvvwesGTVoKs5n95y9GqgNRNotcYbA@mail.gmail.com/3-v4-0002-ci-Simplify-ci-os-only-handling.patch)
  download | inline diff:
From f83dc32773290bfd5ccdade259d810adf2cfc456 Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Fri, 12 Apr 2024 18:18:34 -0700
Subject: [PATCH v4 2/2] ci: Simplify ci-os-only handling

Handle 'ci-os-only' occurrences in the .cirrus.star file instead of
.cirrus.tasks.yml file. Now, 'ci-os-only' occurrences are controlled
from one central place instead of dealing with them in each task.

Author: Andres Freund <[email protected]>
Reviewed-by: Nazir Bilal Yavuz <[email protected]>
Discussion: https://postgr.es/m/20240413021221.hg53rvqlvldqh57i%40awork3.anarazel.de
---
 .cirrus.star      | 31 ++++++++++++++++++++++++++++++-
 .cirrus.tasks.yml | 21 ++++++++++-----------
 2 files changed, 40 insertions(+), 12 deletions(-)

diff --git a/.cirrus.star b/.cirrus.star
index 218fce887b5..83e93e48d5f 100644
--- a/.cirrus.star
+++ b/.cirrus.star
@@ -7,7 +7,7 @@ https://github.com/bazelbuild/starlark/blob/master/spec.md
 See also .cirrus.yml and src/tools/ci/README
 """
 
-load("cirrus", "env", "fs", "yaml")
+load("cirrus", "env", "fs", "re", "yaml")
 
 
 def main():
@@ -66,6 +66,7 @@ def main():
 def compute_environment_vars():
     cenv = {}
 
+    ###
     # The following tasks are manually triggered by default because they
     # might use too many resources for users of free Cirrus credits, but you can
     # trigger them automatically by naming them in an environment variable eg
@@ -82,6 +83,34 @@ def compute_environment_vars():
         else:
             value = 'manual'
         cenv[name] = value
+    ###
+
+    ###
+    # Parse "ci-os-only:" tag in commit message and set
+    # CI_{$OS}_ENABLED variable for each OS
+
+    # We want to disable SanityCheck if testing just a specific OS. This
+    # shortens push-wait-for-ci cycle time a bit when debugging operating
+    # system specific failures. Just treating it as an OS in that case
+    # suffices.
+
+    operating_systems = ['freebsd', 'netbsd', 'openbsd', 'linux', 'macos',
+      'windows', 'mingw', 'sanitycheck', 'compilerwarnings']
+    commit_message = env.get('CIRRUS_CHANGE_MESSAGE')
+    match_re = r"(^|.*\n)ci-os-only: ([^\n]+)($|\n.*)"
+
+    # re.match() returns an array with a tuple of (matched-string, match_1, ...)
+    m = re.match(match_re, commit_message)
+    if m and len(m) > 0:
+        os_only = m[0][2]
+        os_only_list = re.split(r'[, ]+', os_only)
+    else:
+        os_only_list = operating_systems
+
+    for os in operating_systems:
+        os_enabled = os in os_only_list
+        cenv['CI_{0}_ENABLED'.format(os.upper())] = os_enabled
+    ###
 
     return cenv
 
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index f5ff0ccf222..fbaf074b13c 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -62,7 +62,7 @@ task:
   # push-wait-for-ci cycle time a bit when debugging operating system specific
   # failures. Uses skip instead of only_if, as cirrus otherwise warns about
   # only_if conditions not matching.
-  skip: $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:.*'
+  skip: $CI_SANITYCHECK_ENABLED == false
 
   env:
     CPUS: 4
@@ -145,7 +145,7 @@ task:
   <<: *freebsd_task_template
 
   depends_on: SanityCheck
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*freebsd.*'
+  only_if: $CI_FREEBSD_ENABLED
 
   sysinfo_script: |
     id
@@ -237,7 +237,7 @@ task:
     - name: NetBSD - Meson
       # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
       trigger_type: $CI_TRIGGER_TYPE_NETBSD
-      only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*netbsd.*'
+      only_if: $CI_NETBSD_ENABLED
       env:
         OS_NAME: netbsd
         IMAGE_FAMILY: pg-ci-netbsd-postgres
@@ -256,7 +256,7 @@ task:
     - name: OpenBSD - Meson
       # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
       trigger_type: $CI_TRIGGER_TYPE_OPENBSD
-      only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*openbsd.*'
+      only_if: $CI_OPENBSD_ENABLED
       env:
         OS_NAME: openbsd
         IMAGE_FAMILY: pg-ci-openbsd-postgres
@@ -395,7 +395,7 @@ task:
   <<: *linux_task_template
 
   depends_on: SanityCheck
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*'
+  only_if: $CI_LINUX_ENABLED
 
   ccache_cache:
     folder: ${CCACHE_DIR}
@@ -568,7 +568,7 @@ task:
   <<: *macos_task_template
 
   depends_on: SanityCheck
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*'
+  only_if: $CI_MACOS_ENABLED
 
   sysinfo_script: |
     id
@@ -674,7 +674,7 @@ task:
   <<: *windows_task_template
 
   depends_on: SanityCheck
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*'
+  only_if: $CI_WINDOWS_ENABLED
 
   setup_additional_packages_script: |
     REM choco install -y --no-progress ...
@@ -713,7 +713,7 @@ task:
   trigger_type: $CI_TRIGGER_TYPE_MINGW
 
   depends_on: SanityCheck
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
+  only_if: $CI_MINGW_ENABLED
 
   env:
     TEST_JOBS: 4 # higher concurrency causes occasional failures
@@ -767,10 +767,9 @@ task:
 
   # To limit unnecessary work only run this once the SanityCheck
   # succeeds. This is particularly important for this task as we intentionally
-  # use always: to continue after failures. Task that did not run count as a
-  # success, so we need to recheck SanityChecks's condition here ...
+  # use always: to continue after failures.
   depends_on: SanityCheck
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*'
+  only_if: $CI_COMPILERWARNINGS_ENABLED
 
   env:
     CPUS: 4
-- 
2.47.2



^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* Re: ci: Allow running mingw tests by default via environment variable
  2024-04-25 12:02 Re: ci: Allow running mingw tests by default via environment variable Nazir Bilal Yavuz <[email protected]>
  2025-03-04 00:27 ` Re: ci: Allow running mingw tests by default via environment variable Thomas Munro <[email protected]>
  2025-03-04 08:53   ` Re: ci: Allow running mingw tests by default via environment variable Nazir Bilal Yavuz <[email protected]>
@ 2025-03-04 13:00     ` Andres Freund <[email protected]>
  2025-03-05 15:51       ` Re: ci: Allow running mingw tests by default via environment variable Andres Freund <[email protected]>
  0 siblings, 1 reply; 30+ messages in thread

From: Andres Freund @ 2025-03-04 13:00 UTC (permalink / raw)
  To: Nazir Bilal Yavuz <[email protected]>; +Cc: Thomas Munro <[email protected]>; pgsql-hackers; Wolfgang Walther <[email protected]>

Hi,

On 2025-03-04 11:53:30 +0300, Nazir Bilal Yavuz wrote:
> I rebased Andres' v2-0002 on top of recent changes and wrote a small
> commit message. v4 is attached.

Thanks for doing that, it does look a lot better after, I think!

Greetings,

Andres Freund






^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* Re: ci: Allow running mingw tests by default via environment variable
  2024-04-25 12:02 Re: ci: Allow running mingw tests by default via environment variable Nazir Bilal Yavuz <[email protected]>
  2025-03-04 00:27 ` Re: ci: Allow running mingw tests by default via environment variable Thomas Munro <[email protected]>
  2025-03-04 08:53   ` Re: ci: Allow running mingw tests by default via environment variable Nazir Bilal Yavuz <[email protected]>
  2025-03-04 13:00     ` Re: ci: Allow running mingw tests by default via environment variable Andres Freund <[email protected]>
@ 2025-03-05 15:51       ` Andres Freund <[email protected]>
  2025-04-07 13:03         ` Re: ci: Allow running mingw tests by default via environment variable Nazir Bilal Yavuz <[email protected]>
  0 siblings, 1 reply; 30+ messages in thread

From: Andres Freund @ 2025-03-05 15:51 UTC (permalink / raw)
  To: Nazir Bilal Yavuz <[email protected]>; +Cc: Thomas Munro <[email protected]>; pgsql-hackers; Wolfgang Walther <[email protected]>

Hi,

I'm inclined to think we should apply to this to all branches with CI support,
not just master. It's kind of annoying to have CI infrastructure changes like
this that differ between branches.  Thoughts?

Greetings,

Andres Freund






^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* Re: ci: Allow running mingw tests by default via environment variable
  2024-04-25 12:02 Re: ci: Allow running mingw tests by default via environment variable Nazir Bilal Yavuz <[email protected]>
  2025-03-04 00:27 ` Re: ci: Allow running mingw tests by default via environment variable Thomas Munro <[email protected]>
  2025-03-04 08:53   ` Re: ci: Allow running mingw tests by default via environment variable Nazir Bilal Yavuz <[email protected]>
  2025-03-04 13:00     ` Re: ci: Allow running mingw tests by default via environment variable Andres Freund <[email protected]>
  2025-03-05 15:51       ` Re: ci: Allow running mingw tests by default via environment variable Andres Freund <[email protected]>
@ 2025-04-07 13:03         ` Nazir Bilal Yavuz <[email protected]>
  0 siblings, 0 replies; 30+ messages in thread

From: Nazir Bilal Yavuz @ 2025-04-07 13:03 UTC (permalink / raw)
  To: Andres Freund <[email protected]>; +Cc: Thomas Munro <[email protected]>; pgsql-hackers; Wolfgang Walther <[email protected]>

Hi,

On Wed, 5 Mar 2025 at 18:51, Andres Freund <[email protected]> wrote:
>
> Hi,
>
> I'm inclined to think we should apply to this to all branches with CI support,
> not just master. It's kind of annoying to have CI infrastructure changes like
> this that differ between branches.  Thoughts?

I am sharing patches per branch until we come up with a better
solution. Some remarks:

REL_15: It doesn't have any manual tasks, so per repository trigger
patch (Thomas' patch) is omitted.
REL_16: Only Mingw task is manual.
REL_17: Only Mingw task is manual.

While rebasing them, I realized that we use
'REPO_CI_AUTOMATIC_TRIGGER_TASKS' for all branches. So, we don't have
per branch 'REPO_CI_AUTOMATIC_TRIGGER_TASKS' configuration option. Is
that a problem?

-- 
Regards,
Nazir Bilal Yavuz
Microsoft


Attachments:

  [text/x-patch] REL_15_v5-0001-ci-Simplify-ci-os-only-handling.patch (5.5K, ../../CAN55FZ0x55wCTjnPeaNoRr14QkamVfY9UPSdatXqow529ziW5g@mail.gmail.com/2-REL_15_v5-0001-ci-Simplify-ci-os-only-handling.patch)
  download | inline diff:
From c872f537375416289d3fdc302716403266899c1f Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Fri, 12 Apr 2024 18:18:34 -0700
Subject: [PATCH v5] ci: Simplify ci-os-only handling

Handle 'ci-os-only' occurrences in the .cirrus.star file instead of
.cirrus.tasks.yml file. Now, 'ci-os-only' occurrences are controlled
from one central place instead of dealing with them in each task.

Author: Andres Freund <[email protected]>
Reviewed-by: Nazir Bilal Yavuz <[email protected]>
Discussion: https://postgr.es/m/20240413021221.hg53rvqlvldqh57i%40awork3.anarazel.de
Backpatch: 15-, where CI support was added
---
 .cirrus.star      | 52 +++++++++++++++++++++++++++++++++++++++++++----
 .cirrus.tasks.yml | 10 ++++-----
 .cirrus.yml       | 12 +++++++++--
 3 files changed, 63 insertions(+), 11 deletions(-)

diff --git a/.cirrus.star b/.cirrus.star
index d2d6ceca207..3264ce014c5 100644
--- a/.cirrus.star
+++ b/.cirrus.star
@@ -7,7 +7,7 @@ https://github.com/bazelbuild/starlark/blob/master/spec.md
 See also .cirrus.yml and src/tools/ci/README
 """
 
-load("cirrus", "env", "fs")
+load("cirrus", "env", "fs", "re", "yaml")
 
 
 def main():
@@ -18,19 +18,36 @@ def main():
 
     1) the contents of .cirrus.yml
 
-    2) if defined, the contents of the file referenced by the, repository
+    2) computed environment variables
+
+    3) if defined, the contents of the file referenced by the, repository
        level, REPO_CI_CONFIG_GIT_URL variable (see
        https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
        format)
 
-    3) .cirrus.tasks.yml
+    4) .cirrus.tasks.yml
     """
 
     output = ""
 
     # 1) is evaluated implicitly
 
+
     # Add 2)
+    additional_env = compute_environment_vars()
+    env_fmt = """
+###
+# Computed environment variables start here
+###
+{0}
+###
+# Computed environment variables end here
+###
+"""
+    output += env_fmt.format(yaml.dumps({'env': additional_env}))
+
+
+    # Add 3)
     repo_config_url = env.get("REPO_CI_CONFIG_GIT_URL")
     if repo_config_url != None:
         print("loading additional configuration from \"{}\"".format(repo_config_url))
@@ -38,12 +55,39 @@ def main():
     else:
         output += "\n# REPO_CI_CONFIG_URL was not set\n"
 
-    # Add 3)
+
+    # Add 4)
     output += config_from(".cirrus.tasks.yml")
 
+
     return output
 
 
+def compute_environment_vars():
+    cenv = {}
+
+    # Parse "ci-os-only:" tag in commit message and set
+    # CI_{$OS}_ENABLED variable for each OS
+
+    operating_systems = ['freebsd', 'linux', 'macos', 'windows', 'compilerwarnings']
+    commit_message = env.get('CIRRUS_CHANGE_MESSAGE')
+    match_re = r"(^|.*\n)ci-os-only: ([^\n]+)($|\n.*)"
+
+    # re.match() returns an array with a tuple of (matched-string, match_1, ...)
+    m = re.match(match_re, commit_message)
+    if m and len(m) > 0:
+        os_only = m[0][2]
+        os_only_list = re.split(r'[, ]+', os_only)
+    else:
+        os_only_list = operating_systems
+
+    for os in operating_systems:
+        os_enabled = os in os_only_list
+        cenv['CI_{0}_ENABLED'.format(os.upper())] = os_enabled
+
+    return cenv
+
+
 def config_from(config_src):
     """return contents of config file `config_src`, surrounded by markers
     indicating start / end of the the included file
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 995ae7cefb9..ecf683b7b54 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -47,7 +47,7 @@ task:
 
   <<: *freebsd_task_template
 
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*freebsd.*'
+  only_if: $CI_FREEBSD_ENABLED
 
   sysinfo_script: |
     id
@@ -153,7 +153,7 @@ task:
 
   <<: *linux_task_template
 
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*'
+  only_if: $CI_LINUX_ENABLED
 
   ccache_cache:
     folder: ${CCACHE_DIR}
@@ -239,7 +239,7 @@ task:
 
   <<: *macos_task_template
 
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*'
+  only_if: $CI_MACOS_ENABLED
 
   sysinfo_script: |
     id
@@ -389,7 +389,7 @@ task:
 
   <<: *windows_task_template
 
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*'
+  only_if: $CI_WINDOWS_ENABLED
 
   sysinfo_script: |
     chcp
@@ -476,7 +476,7 @@ task:
 
   # task that did not run, count as a success, so we need to recheck Linux'
   # condition here ...
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*'
+  only_if: $CI_COMPILERWARNINGS_ENABLED
 
   <<: *linux_task_template
 
diff --git a/.cirrus.yml b/.cirrus.yml
index a83129ae46d..f270f61241f 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -10,12 +10,20 @@
 #
 # 1) the contents of this file
 #
-# 2) if defined, the contents of the file referenced by the, repository
+# 2) computed environment variables
+#
+#    Used to enable/disable tasks based on the execution environment. See
+#    .cirrus.star: compute_environment_vars()
+#
+# 3) if defined, the contents of the file referenced by the, repository
 #    level, REPO_CI_CONFIG_GIT_URL variable (see
 #    https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
 #    format)
 #
-# 3) .cirrus.tasks.yml
+#    This allows running tasks in a different execution environment than the
+#    default, e.g. to have sufficient resources for cfbot.
+#
+# 4) .cirrus.tasks.yml
 #
 # This composition is done by .cirrus.star
 
-- 
2.49.0



  [text/x-patch] REL_16_v5-0001-ci-Per-repo-triggers-for-not-automated-tasks.patch (6.3K, ../../CAN55FZ0x55wCTjnPeaNoRr14QkamVfY9UPSdatXqow529ziW5g@mail.gmail.com/3-REL_16_v5-0001-ci-Per-repo-triggers-for-not-automated-tasks.patch)
  download | inline diff:
From b9276d7cc5713008a986c68d2f00b5bf23cb50fd Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Fri, 12 Apr 2024 15:04:32 -0700
Subject: [PATCH v5 1/2] ci: Per-repo triggers for not automated tasks

We're not ready to trigger some tasks automatically by default, so you
normally have to trigger then manually via Cirrus if you want to run
them.  This avoids using too many credits by default.

With this commit, an individual repository can be configured to trigger
them automatically using an environment variable defined under
"Repository Settings", for example:

REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw"

This will enable cfbot to turn them on by default when running tests for
the Commitfest app.  It is not subject to free credit limits as it runs
on credits donated by Google.

Author: Andres Freund <[email protected]>
Co-authored-by: Thomas Munro <[email protected]>
Reviewed-by: Nazir Bilal Yavuz <[email protected]>
Discussion: https://postgr.es/m/20240413021221.hg53rvqlvldqh57i%40awork3.anarazel.de
Backpatch: 16-, where first manual task was added
---
 .cirrus.star        | 50 +++++++++++++++++++++++++++++++++++++++++----
 .cirrus.tasks.yml   | 10 ++++-----
 .cirrus.yml         | 12 +++++++++--
 src/tools/ci/README | 11 ++++++++++
 4 files changed, 71 insertions(+), 12 deletions(-)

diff --git a/.cirrus.star b/.cirrus.star
index d2d6ceca207..ff31d191565 100644
--- a/.cirrus.star
+++ b/.cirrus.star
@@ -7,7 +7,7 @@ https://github.com/bazelbuild/starlark/blob/master/spec.md
 See also .cirrus.yml and src/tools/ci/README
 """
 
-load("cirrus", "env", "fs")
+load("cirrus", "env", "fs", "yaml")
 
 
 def main():
@@ -18,19 +18,36 @@ def main():
 
     1) the contents of .cirrus.yml
 
-    2) if defined, the contents of the file referenced by the, repository
+    2) computed environment variables
+
+    3) if defined, the contents of the file referenced by the, repository
        level, REPO_CI_CONFIG_GIT_URL variable (see
        https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
        format)
 
-    3) .cirrus.tasks.yml
+    4) .cirrus.tasks.yml
     """
 
     output = ""
 
     # 1) is evaluated implicitly
 
+
     # Add 2)
+    additional_env = compute_environment_vars()
+    env_fmt = """
+###
+# Computed environment variables start here
+###
+{0}
+###
+# Computed environment variables end here
+###
+"""
+    output += env_fmt.format(yaml.dumps({'env': additional_env}))
+
+
+    # Add 3)
     repo_config_url = env.get("REPO_CI_CONFIG_GIT_URL")
     if repo_config_url != None:
         print("loading additional configuration from \"{}\"".format(repo_config_url))
@@ -38,12 +55,37 @@ def main():
     else:
         output += "\n# REPO_CI_CONFIG_URL was not set\n"
 
-    # Add 3)
+
+    # Add 4)
     output += config_from(".cirrus.tasks.yml")
 
+
     return output
 
 
+def compute_environment_vars():
+    cenv = {}
+
+    # The mingw task is manually triggered by default because it might use too
+    # many resources for users of free Cirrus credits, but you can trigger it
+    # automatically by naming it in an environment variable eg
+    # REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw" under "Repository Settings" on
+    # Cirrus CI's website.
+
+    default_manual_trigger_tasks = ['mingw']
+
+    repo_ci_automatic_trigger_tasks = env.get('REPO_CI_AUTOMATIC_TRIGGER_TASKS', '')
+    for task in default_manual_trigger_tasks:
+        name = 'CI_TRIGGER_TYPE_' + task.upper()
+        if repo_ci_automatic_trigger_tasks.find(task) != -1:
+            value = 'automatic'
+        else:
+            value = 'manual'
+        cenv[name] = value
+
+    return cenv
+
+
 def config_from(config_src):
     """return contents of config file `config_src`, surrounded by markers
     indicating start / end of the the included file
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index d12360aa356..54c6f3a65ee 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -591,13 +591,11 @@ task:
   << : *WINDOWS_ENVIRONMENT_BASE
   name: Windows - Server 2019, MinGW64 - Meson
 
-  # due to resource constraints we don't run this task by default for now
-  trigger_type: manual
-  # worth using only_if despite being manual, otherwise this task will show up
-  # when e.g. ci-os-only: linux is used.
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
-  # otherwise it'll be sorted before other tasks
+  # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star.
+  trigger_type: $CI_TRIGGER_TYPE_MINGW
+
   depends_on: SanityCheck
+  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
 
   env:
     TEST_JOBS: 4 # higher concurrency causes occasional failures
diff --git a/.cirrus.yml b/.cirrus.yml
index a83129ae46d..f270f61241f 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -10,12 +10,20 @@
 #
 # 1) the contents of this file
 #
-# 2) if defined, the contents of the file referenced by the, repository
+# 2) computed environment variables
+#
+#    Used to enable/disable tasks based on the execution environment. See
+#    .cirrus.star: compute_environment_vars()
+#
+# 3) if defined, the contents of the file referenced by the, repository
 #    level, REPO_CI_CONFIG_GIT_URL variable (see
 #    https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
 #    format)
 #
-# 3) .cirrus.tasks.yml
+#    This allows running tasks in a different execution environment than the
+#    default, e.g. to have sufficient resources for cfbot.
+#
+# 4) .cirrus.tasks.yml
 #
 # This composition is done by .cirrus.star
 
diff --git a/src/tools/ci/README b/src/tools/ci/README
index 30ddd200c96..771243c4da0 100644
--- a/src/tools/ci/README
+++ b/src/tools/ci/README
@@ -82,3 +82,14 @@ defined in .cirrus.yml, by redefining the relevant yaml anchors.
 Custom compute resources can be provided using
 - https://cirrus-ci.org/guide/supported-computing-services/
 - https://cirrus-ci.org/guide/persistent-workers/
+
+
+Enabling manual tasks by default
+================================
+
+Some tasks are not triggered automatically by default, to avoid using up CI
+credits too quickly. This can be changed on the repository level, e.g. when
+custom compute resources are configured.
+
+The following repository level environment variables are recognized:
+- REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw"
-- 
2.49.0



  [text/x-patch] REL_16_v5-0002-ci-Simplify-ci-os-only-handling.patch (5.0K, ../../CAN55FZ0x55wCTjnPeaNoRr14QkamVfY9UPSdatXqow529ziW5g@mail.gmail.com/4-REL_16_v5-0002-ci-Simplify-ci-os-only-handling.patch)
  download | inline diff:
From 5975233e10c8bb4e87049cccb13c98cbdc342532 Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Fri, 12 Apr 2024 18:18:34 -0700
Subject: [PATCH v5 2/2] ci: Simplify ci-os-only handling

Handle 'ci-os-only' occurrences in the .cirrus.star file instead of
.cirrus.tasks.yml file. Now, 'ci-os-only' occurrences are controlled
from one central place instead of dealing with them in each task.

Author: Andres Freund <[email protected]>
Reviewed-by: Nazir Bilal Yavuz <[email protected]>
Discussion: https://postgr.es/m/20240413021221.hg53rvqlvldqh57i%40awork3.anarazel.de
Backpatch: 15-, where CI support was added
---
 .cirrus.star      | 31 ++++++++++++++++++++++++++++++-
 .cirrus.tasks.yml | 17 ++++++++---------
 2 files changed, 38 insertions(+), 10 deletions(-)

diff --git a/.cirrus.star b/.cirrus.star
index ff31d191565..06666e55a4b 100644
--- a/.cirrus.star
+++ b/.cirrus.star
@@ -7,7 +7,7 @@ https://github.com/bazelbuild/starlark/blob/master/spec.md
 See also .cirrus.yml and src/tools/ci/README
 """
 
-load("cirrus", "env", "fs", "yaml")
+load("cirrus", "env", "fs", "re", "yaml")
 
 
 def main():
@@ -66,6 +66,7 @@ def main():
 def compute_environment_vars():
     cenv = {}
 
+    ###
     # The mingw task is manually triggered by default because it might use too
     # many resources for users of free Cirrus credits, but you can trigger it
     # automatically by naming it in an environment variable eg
@@ -82,6 +83,34 @@ def compute_environment_vars():
         else:
             value = 'manual'
         cenv[name] = value
+    ###
+
+    ###
+    # Parse "ci-os-only:" tag in commit message and set
+    # CI_{$OS}_ENABLED variable for each OS
+
+    # We want to disable SanityCheck if testing just a specific OS. This
+    # shortens push-wait-for-ci cycle time a bit when debugging operating
+    # system specific failures. Just treating it as an OS in that case
+    # suffices.
+
+    operating_systems = ['freebsd', 'linux', 'macos', 'windows', 'mingw',
+      'sanitycheck', 'compilerwarnings']
+    commit_message = env.get('CIRRUS_CHANGE_MESSAGE')
+    match_re = r"(^|.*\n)ci-os-only: ([^\n]+)($|\n.*)"
+
+    # re.match() returns an array with a tuple of (matched-string, match_1, ...)
+    m = re.match(match_re, commit_message)
+    if m and len(m) > 0:
+        os_only = m[0][2]
+        os_only_list = re.split(r'[, ]+', os_only)
+    else:
+        os_only_list = operating_systems
+
+    for os in operating_systems:
+        os_enabled = os in os_only_list
+        cenv['CI_{0}_ENABLED'.format(os.upper())] = os_enabled
+    ###
 
     return cenv
 
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 54c6f3a65ee..59d69acf2a7 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -62,7 +62,7 @@ task:
   # push-wait-for-ci cycle time a bit when debugging operating system specific
   # failures. Uses skip instead of only_if, as cirrus otherwise warns about
   # only_if conditions not matching.
-  skip: $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:.*'
+  skip: $CI_SANITYCHECK_ENABLED == false
 
   env:
     CPUS: 4
@@ -143,7 +143,7 @@ task:
   <<: *freebsd_task_template
 
   depends_on: SanityCheck
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*freebsd.*'
+  only_if: $CI_FREEBSD_ENABLED
 
   sysinfo_script: |
     id
@@ -280,7 +280,7 @@ task:
   <<: *linux_task_template
 
   depends_on: SanityCheck
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*'
+  only_if: $CI_LINUX_ENABLED
 
   ccache_cache:
     folder: ${CCACHE_DIR}
@@ -449,7 +449,7 @@ task:
   <<: *macos_task_template
 
   depends_on: SanityCheck
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*'
+  only_if: $CI_MACOS_ENABLED
 
   sysinfo_script: |
     id
@@ -556,7 +556,7 @@ task:
   <<: *windows_task_template
 
   depends_on: SanityCheck
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*'
+  only_if: $CI_WINDOWS_ENABLED
 
   setup_additional_packages_script: |
     REM choco install -y --no-progress ...
@@ -595,7 +595,7 @@ task:
   trigger_type: $CI_TRIGGER_TYPE_MINGW
 
   depends_on: SanityCheck
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
+  only_if: $CI_MINGW_ENABLED
 
   env:
     TEST_JOBS: 4 # higher concurrency causes occasional failures
@@ -649,10 +649,9 @@ task:
 
   # To limit unnecessary work only run this once the SanityCheck
   # succeeds. This is particularly important for this task as we intentionally
-  # use always: to continue after failures. Task that did not run count as a
-  # success, so we need to recheck SanityChecks's condition here ...
+  # use always: to continue after failures.
   depends_on: SanityCheck
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*'
+  only_if: $CI_COMPILERWARNINGS_ENABLED
 
   env:
     CPUS: 4
-- 
2.49.0



  [text/x-patch] REL_17_v5-0001-ci-Per-repo-triggers-for-not-automated-tasks.patch (6.3K, ../../CAN55FZ0x55wCTjnPeaNoRr14QkamVfY9UPSdatXqow529ziW5g@mail.gmail.com/5-REL_17_v5-0001-ci-Per-repo-triggers-for-not-automated-tasks.patch)
  download | inline diff:
From 340782a5e770b85d05f4e583d6dc1f12b1888a96 Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Fri, 12 Apr 2024 15:04:32 -0700
Subject: [PATCH v5 1/2] ci: Per-repo triggers for not automated tasks

We're not ready to trigger some tasks automatically by default, so you
normally have to trigger then manually via Cirrus if you want to run
them.  This avoids using too many credits by default.

With this commit, an individual repository can be configured to trigger
them automatically using an environment variable defined under
"Repository Settings", for example:

REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw"

This will enable cfbot to turn them on by default when running tests for
the Commitfest app.  It is not subject to free credit limits as it runs
on credits donated by Google.

Author: Andres Freund <[email protected]>
Co-authored-by: Thomas Munro <[email protected]>
Reviewed-by: Nazir Bilal Yavuz <[email protected]>
Discussion: https://postgr.es/m/20240413021221.hg53rvqlvldqh57i%40awork3.anarazel.de
Backpatch: 16-, where first manual task was added
---
 .cirrus.star        | 50 +++++++++++++++++++++++++++++++++++++++++----
 .cirrus.tasks.yml   | 10 ++++-----
 .cirrus.yml         | 12 +++++++++--
 src/tools/ci/README | 11 ++++++++++
 4 files changed, 71 insertions(+), 12 deletions(-)

diff --git a/.cirrus.star b/.cirrus.star
index 36233872d1e..cccdbd1bfbb 100644
--- a/.cirrus.star
+++ b/.cirrus.star
@@ -7,7 +7,7 @@ https://github.com/bazelbuild/starlark/blob/master/spec.md
 See also .cirrus.yml and src/tools/ci/README
 """
 
-load("cirrus", "env", "fs")
+load("cirrus", "env", "fs", "yaml")
 
 
 def main():
@@ -18,19 +18,36 @@ def main():
 
     1) the contents of .cirrus.yml
 
-    2) if defined, the contents of the file referenced by the, repository
+    2) computed environment variables
+
+    3) if defined, the contents of the file referenced by the, repository
        level, REPO_CI_CONFIG_GIT_URL variable (see
        https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
        format)
 
-    3) .cirrus.tasks.yml
+    4) .cirrus.tasks.yml
     """
 
     output = ""
 
     # 1) is evaluated implicitly
 
+
     # Add 2)
+    additional_env = compute_environment_vars()
+    env_fmt = """
+###
+# Computed environment variables start here
+###
+{0}
+###
+# Computed environment variables end here
+###
+"""
+    output += env_fmt.format(yaml.dumps({'env': additional_env}))
+
+
+    # Add 3)
     repo_config_url = env.get("REPO_CI_CONFIG_GIT_URL")
     if repo_config_url != None:
         print("loading additional configuration from \"{}\"".format(repo_config_url))
@@ -38,12 +55,37 @@ def main():
     else:
         output += "\n# REPO_CI_CONFIG_URL was not set\n"
 
-    # Add 3)
+
+    # Add 4)
     output += config_from(".cirrus.tasks.yml")
 
+
     return output
 
 
+def compute_environment_vars():
+    cenv = {}
+
+    # The mingw task is manually triggered by default because it might use too
+    # many resources for users of free Cirrus credits, but you can trigger it
+    # automatically by naming it in an environment variable eg
+    # REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw" under "Repository Settings" on
+    # Cirrus CI's website.
+
+    default_manual_trigger_tasks = ['mingw']
+
+    repo_ci_automatic_trigger_tasks = env.get('REPO_CI_AUTOMATIC_TRIGGER_TASKS', '')
+    for task in default_manual_trigger_tasks:
+        name = 'CI_TRIGGER_TYPE_' + task.upper()
+        if repo_ci_automatic_trigger_tasks.find(task) != -1:
+            value = 'automatic'
+        else:
+            value = 'manual'
+        cenv[name] = value
+
+    return cenv
+
+
 def config_from(config_src):
     """return contents of config file `config_src`, surrounded by markers
     indicating start / end of the included file
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index aa7c7dcd34d..af33716a004 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -595,13 +595,11 @@ task:
   << : *WINDOWS_ENVIRONMENT_BASE
   name: Windows - Server 2019, MinGW64 - Meson
 
-  # due to resource constraints we don't run this task by default for now
-  trigger_type: manual
-  # worth using only_if despite being manual, otherwise this task will show up
-  # when e.g. ci-os-only: linux is used.
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
-  # otherwise it'll be sorted before other tasks
+  # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star.
+  trigger_type: $CI_TRIGGER_TYPE_MINGW
+
   depends_on: SanityCheck
+  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
 
   env:
     TEST_JOBS: 4 # higher concurrency causes occasional failures
diff --git a/.cirrus.yml b/.cirrus.yml
index a83129ae46d..f270f61241f 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -10,12 +10,20 @@
 #
 # 1) the contents of this file
 #
-# 2) if defined, the contents of the file referenced by the, repository
+# 2) computed environment variables
+#
+#    Used to enable/disable tasks based on the execution environment. See
+#    .cirrus.star: compute_environment_vars()
+#
+# 3) if defined, the contents of the file referenced by the, repository
 #    level, REPO_CI_CONFIG_GIT_URL variable (see
 #    https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
 #    format)
 #
-# 3) .cirrus.tasks.yml
+#    This allows running tasks in a different execution environment than the
+#    default, e.g. to have sufficient resources for cfbot.
+#
+# 4) .cirrus.tasks.yml
 #
 # This composition is done by .cirrus.star
 
diff --git a/src/tools/ci/README b/src/tools/ci/README
index 30ddd200c96..771243c4da0 100644
--- a/src/tools/ci/README
+++ b/src/tools/ci/README
@@ -82,3 +82,14 @@ defined in .cirrus.yml, by redefining the relevant yaml anchors.
 Custom compute resources can be provided using
 - https://cirrus-ci.org/guide/supported-computing-services/
 - https://cirrus-ci.org/guide/persistent-workers/
+
+
+Enabling manual tasks by default
+================================
+
+Some tasks are not triggered automatically by default, to avoid using up CI
+credits too quickly. This can be changed on the repository level, e.g. when
+custom compute resources are configured.
+
+The following repository level environment variables are recognized:
+- REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw"
-- 
2.49.0



  [text/x-patch] REL_17_v5-0002-ci-Simplify-ci-os-only-handling.patch (5.0K, ../../CAN55FZ0x55wCTjnPeaNoRr14QkamVfY9UPSdatXqow529ziW5g@mail.gmail.com/6-REL_17_v5-0002-ci-Simplify-ci-os-only-handling.patch)
  download | inline diff:
From 903489cc02b40f44e3672dda2fdbcd9dd7147056 Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Fri, 12 Apr 2024 18:18:34 -0700
Subject: [PATCH v5 2/2] ci: Simplify ci-os-only handling

Handle 'ci-os-only' occurrences in the .cirrus.star file instead of
.cirrus.tasks.yml file. Now, 'ci-os-only' occurrences are controlled
from one central place instead of dealing with them in each task.

Author: Andres Freund <[email protected]>
Reviewed-by: Nazir Bilal Yavuz <[email protected]>
Discussion: https://postgr.es/m/20240413021221.hg53rvqlvldqh57i%40awork3.anarazel.de
Backpatch: 15-, where CI support was added
---
 .cirrus.star      | 31 ++++++++++++++++++++++++++++++-
 .cirrus.tasks.yml | 17 ++++++++---------
 2 files changed, 38 insertions(+), 10 deletions(-)

diff --git a/.cirrus.star b/.cirrus.star
index cccdbd1bfbb..c26cf52d10c 100644
--- a/.cirrus.star
+++ b/.cirrus.star
@@ -7,7 +7,7 @@ https://github.com/bazelbuild/starlark/blob/master/spec.md
 See also .cirrus.yml and src/tools/ci/README
 """
 
-load("cirrus", "env", "fs", "yaml")
+load("cirrus", "env", "fs", "re", "yaml")
 
 
 def main():
@@ -66,6 +66,7 @@ def main():
 def compute_environment_vars():
     cenv = {}
 
+    ###
     # The mingw task is manually triggered by default because it might use too
     # many resources for users of free Cirrus credits, but you can trigger it
     # automatically by naming it in an environment variable eg
@@ -82,6 +83,34 @@ def compute_environment_vars():
         else:
             value = 'manual'
         cenv[name] = value
+    ###
+
+    ###
+    # Parse "ci-os-only:" tag in commit message and set
+    # CI_{$OS}_ENABLED variable for each OS
+
+    # We want to disable SanityCheck if testing just a specific OS. This
+    # shortens push-wait-for-ci cycle time a bit when debugging operating
+    # system specific failures. Just treating it as an OS in that case
+    # suffices.
+
+    operating_systems = ['freebsd', 'linux', 'macos', 'windows', 'mingw',
+      'sanitycheck', 'compilerwarnings']
+    commit_message = env.get('CIRRUS_CHANGE_MESSAGE')
+    match_re = r"(^|.*\n)ci-os-only: ([^\n]+)($|\n.*)"
+
+    # re.match() returns an array with a tuple of (matched-string, match_1, ...)
+    m = re.match(match_re, commit_message)
+    if m and len(m) > 0:
+        os_only = m[0][2]
+        os_only_list = re.split(r'[, ]+', os_only)
+    else:
+        os_only_list = operating_systems
+
+    for os in operating_systems:
+        os_enabled = os in os_only_list
+        cenv['CI_{0}_ENABLED'.format(os.upper())] = os_enabled
+    ###
 
     return cenv
 
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index af33716a004..93459e808f0 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -62,7 +62,7 @@ task:
   # push-wait-for-ci cycle time a bit when debugging operating system specific
   # failures. Uses skip instead of only_if, as cirrus otherwise warns about
   # only_if conditions not matching.
-  skip: $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:.*'
+  skip: $CI_SANITYCHECK_ENABLED == false
 
   env:
     CPUS: 4
@@ -144,7 +144,7 @@ task:
   <<: *freebsd_task_template
 
   depends_on: SanityCheck
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*freebsd.*'
+  only_if: $CI_FREEBSD_ENABLED
 
   sysinfo_script: |
     id
@@ -282,7 +282,7 @@ task:
   <<: *linux_task_template
 
   depends_on: SanityCheck
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*'
+  only_if: $CI_LINUX_ENABLED
 
   ccache_cache:
     folder: ${CCACHE_DIR}
@@ -453,7 +453,7 @@ task:
   <<: *macos_task_template
 
   depends_on: SanityCheck
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*'
+  only_if: $CI_MACOS_ENABLED
 
   sysinfo_script: |
     id
@@ -560,7 +560,7 @@ task:
   <<: *windows_task_template
 
   depends_on: SanityCheck
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*'
+  only_if: $CI_WINDOWS_ENABLED
 
   setup_additional_packages_script: |
     REM choco install -y --no-progress ...
@@ -599,7 +599,7 @@ task:
   trigger_type: $CI_TRIGGER_TYPE_MINGW
 
   depends_on: SanityCheck
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
+  only_if: $CI_MINGW_ENABLED
 
   env:
     TEST_JOBS: 4 # higher concurrency causes occasional failures
@@ -653,10 +653,9 @@ task:
 
   # To limit unnecessary work only run this once the SanityCheck
   # succeeds. This is particularly important for this task as we intentionally
-  # use always: to continue after failures. Task that did not run count as a
-  # success, so we need to recheck SanityChecks's condition here ...
+  # use always: to continue after failures.
   depends_on: SanityCheck
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*'
+  only_if: $CI_COMPILERWARNINGS_ENABLED
 
   env:
     CPUS: 4
-- 
2.49.0



  [text/x-patch] Upstream_v5-0001-ci-Per-repo-triggers-for-not-automated-tasks.patch (7.2K, ../../CAN55FZ0x55wCTjnPeaNoRr14QkamVfY9UPSdatXqow529ziW5g@mail.gmail.com/7-Upstream_v5-0001-ci-Per-repo-triggers-for-not-automated-tasks.patch)
  download | inline diff:
From c50fdccc3b3e1aee9f51c576da5942bcc1732e14 Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Fri, 12 Apr 2024 15:04:32 -0700
Subject: [PATCH v5 1/2] ci: Per-repo triggers for not automated tasks

We're not ready to trigger some tasks automatically by default, so you
normally have to trigger then manually via Cirrus if you want to run
them.  This avoids using too many credits by default.

With this commit, an individual repository can be configured to trigger
them automatically using an environment variable defined under
"Repository Settings", for example:

REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw netbsd openbsd"

This will enable cfbot to turn them on by default when running tests for
the Commitfest app.  It is not subject to free credit limits as it runs
on credits donated by Google.

Author: Andres Freund <[email protected]>
Co-authored-by: Thomas Munro <[email protected]>
Reviewed-by: Nazir Bilal Yavuz <[email protected]>
Discussion: https://postgr.es/m/20240413021221.hg53rvqlvldqh57i%40awork3.anarazel.de
Backpatch: 16-, where first manual task was added
---
 .cirrus.star        | 50 +++++++++++++++++++++++++++++++++++++++++----
 .cirrus.tasks.yml   | 15 +++++++-------
 .cirrus.yml         | 12 +++++++++--
 src/tools/ci/README | 11 ++++++++++
 4 files changed, 75 insertions(+), 13 deletions(-)

diff --git a/.cirrus.star b/.cirrus.star
index 36233872d1e..218fce887b5 100644
--- a/.cirrus.star
+++ b/.cirrus.star
@@ -7,7 +7,7 @@ https://github.com/bazelbuild/starlark/blob/master/spec.md
 See also .cirrus.yml and src/tools/ci/README
 """
 
-load("cirrus", "env", "fs")
+load("cirrus", "env", "fs", "yaml")
 
 
 def main():
@@ -18,19 +18,36 @@ def main():
 
     1) the contents of .cirrus.yml
 
-    2) if defined, the contents of the file referenced by the, repository
+    2) computed environment variables
+
+    3) if defined, the contents of the file referenced by the, repository
        level, REPO_CI_CONFIG_GIT_URL variable (see
        https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
        format)
 
-    3) .cirrus.tasks.yml
+    4) .cirrus.tasks.yml
     """
 
     output = ""
 
     # 1) is evaluated implicitly
 
+
     # Add 2)
+    additional_env = compute_environment_vars()
+    env_fmt = """
+###
+# Computed environment variables start here
+###
+{0}
+###
+# Computed environment variables end here
+###
+"""
+    output += env_fmt.format(yaml.dumps({'env': additional_env}))
+
+
+    # Add 3)
     repo_config_url = env.get("REPO_CI_CONFIG_GIT_URL")
     if repo_config_url != None:
         print("loading additional configuration from \"{}\"".format(repo_config_url))
@@ -38,12 +55,37 @@ def main():
     else:
         output += "\n# REPO_CI_CONFIG_URL was not set\n"
 
-    # Add 3)
+
+    # Add 4)
     output += config_from(".cirrus.tasks.yml")
 
+
     return output
 
 
+def compute_environment_vars():
+    cenv = {}
+
+    # The following tasks are manually triggered by default because they
+    # might use too many resources for users of free Cirrus credits, but you can
+    # trigger them automatically by naming them in an environment variable eg
+    # REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw netbsd" under "Repository
+    # Settings" on Cirrus CI's website.
+
+    default_manual_trigger_tasks = ['mingw', 'netbsd', 'openbsd']
+
+    repo_ci_automatic_trigger_tasks = env.get('REPO_CI_AUTOMATIC_TRIGGER_TASKS', '')
+    for task in default_manual_trigger_tasks:
+        name = 'CI_TRIGGER_TYPE_' + task.upper()
+        if repo_ci_automatic_trigger_tasks.find(task) != -1:
+            value = 'automatic'
+        else:
+            value = 'manual'
+        cenv[name] = value
+
+    return cenv
+
+
 def config_from(config_src):
     """return contents of config file `config_src`, surrounded by markers
     indicating start / end of the included file
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 86a1fa9bbdb..0cf6ea67ba5 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -239,7 +239,6 @@ task:
 
 task:
   depends_on: SanityCheck
-  trigger_type: manual
 
   env:
     # Below are experimentally derived to be a decent choice.
@@ -257,6 +256,8 @@ task:
 
   matrix:
     - name: NetBSD - Meson
+      # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
+      trigger_type: $CI_TRIGGER_TYPE_NETBSD
       only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*netbsd.*'
       env:
         OS_NAME: netbsd
@@ -274,6 +275,8 @@ task:
       <<: *netbsd_task_template
 
     - name: OpenBSD - Meson
+      # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
+      trigger_type: $CI_TRIGGER_TYPE_OPENBSD
       only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*openbsd.*'
       env:
         OS_NAME: openbsd
@@ -743,13 +746,11 @@ task:
   << : *WINDOWS_ENVIRONMENT_BASE
   name: Windows - Server 2019, MinGW64 - Meson
 
-  # due to resource constraints we don't run this task by default for now
-  trigger_type: manual
-  # worth using only_if despite being manual, otherwise this task will show up
-  # when e.g. ci-os-only: linux is used.
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
-  # otherwise it'll be sorted before other tasks
+  # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star.
+  trigger_type: $CI_TRIGGER_TYPE_MINGW
+
   depends_on: SanityCheck
+  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
 
   env:
     TEST_JOBS: 4 # higher concurrency causes occasional failures
diff --git a/.cirrus.yml b/.cirrus.yml
index 33c6e481d74..3f75852e84e 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -10,12 +10,20 @@
 #
 # 1) the contents of this file
 #
-# 2) if defined, the contents of the file referenced by the, repository
+# 2) computed environment variables
+#
+#    Used to enable/disable tasks based on the execution environment. See
+#    .cirrus.star: compute_environment_vars()
+#
+# 3) if defined, the contents of the file referenced by the, repository
 #    level, REPO_CI_CONFIG_GIT_URL variable (see
 #    https://cirrus-ci.org/guide/programming-tasks/#fs for the accepted
 #    format)
 #
-# 3) .cirrus.tasks.yml
+#    This allows running tasks in a different execution environment than the
+#    default, e.g. to have sufficient resources for cfbot.
+#
+# 4) .cirrus.tasks.yml
 #
 # This composition is done by .cirrus.star
 
diff --git a/src/tools/ci/README b/src/tools/ci/README
index 12c1e7c308f..d183648a8d0 100644
--- a/src/tools/ci/README
+++ b/src/tools/ci/README
@@ -82,3 +82,14 @@ defined in .cirrus.yml, by redefining the relevant yaml anchors.
 Custom compute resources can be provided using
 - https://cirrus-ci.org/guide/supported-computing-services/
 - https://cirrus-ci.org/guide/persistent-workers/
+
+
+Enabling manual tasks by default
+================================
+
+Some tasks are not triggered automatically by default, to avoid using up CI
+credits too quickly. This can be changed on the repository level, e.g. when
+custom compute resources are configured.
+
+The following repository level environment variables are recognized:
+- REPO_CI_AUTOMATIC_TRIGGER_TASKS - space-separated list of (mingw|netbsd|openbsd)
-- 
2.49.0



  [text/x-patch] Upstream_v5-0002-ci-Simplify-ci-os-only-handling.patch (5.8K, ../../CAN55FZ0x55wCTjnPeaNoRr14QkamVfY9UPSdatXqow529ziW5g@mail.gmail.com/8-Upstream_v5-0002-ci-Simplify-ci-os-only-handling.patch)
  download | inline diff:
From bcd64a5fb6d6721fceba1684a3bfc0b46c7fd9d5 Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Fri, 12 Apr 2024 18:18:34 -0700
Subject: [PATCH v5 2/2] ci: Simplify ci-os-only handling

Handle 'ci-os-only' occurrences in the .cirrus.star file instead of
.cirrus.tasks.yml file. Now, 'ci-os-only' occurrences are controlled
from one central place instead of dealing with them in each task.

Author: Andres Freund <[email protected]>
Reviewed-by: Nazir Bilal Yavuz <[email protected]>
Discussion: https://postgr.es/m/20240413021221.hg53rvqlvldqh57i%40awork3.anarazel.de
Backpatch: 15-, where CI support was added
---
 .cirrus.star      | 31 ++++++++++++++++++++++++++++++-
 .cirrus.tasks.yml | 21 ++++++++++-----------
 2 files changed, 40 insertions(+), 12 deletions(-)

diff --git a/.cirrus.star b/.cirrus.star
index 218fce887b5..83e93e48d5f 100644
--- a/.cirrus.star
+++ b/.cirrus.star
@@ -7,7 +7,7 @@ https://github.com/bazelbuild/starlark/blob/master/spec.md
 See also .cirrus.yml and src/tools/ci/README
 """
 
-load("cirrus", "env", "fs", "yaml")
+load("cirrus", "env", "fs", "re", "yaml")
 
 
 def main():
@@ -66,6 +66,7 @@ def main():
 def compute_environment_vars():
     cenv = {}
 
+    ###
     # The following tasks are manually triggered by default because they
     # might use too many resources for users of free Cirrus credits, but you can
     # trigger them automatically by naming them in an environment variable eg
@@ -82,6 +83,34 @@ def compute_environment_vars():
         else:
             value = 'manual'
         cenv[name] = value
+    ###
+
+    ###
+    # Parse "ci-os-only:" tag in commit message and set
+    # CI_{$OS}_ENABLED variable for each OS
+
+    # We want to disable SanityCheck if testing just a specific OS. This
+    # shortens push-wait-for-ci cycle time a bit when debugging operating
+    # system specific failures. Just treating it as an OS in that case
+    # suffices.
+
+    operating_systems = ['freebsd', 'netbsd', 'openbsd', 'linux', 'macos',
+      'windows', 'mingw', 'sanitycheck', 'compilerwarnings']
+    commit_message = env.get('CIRRUS_CHANGE_MESSAGE')
+    match_re = r"(^|.*\n)ci-os-only: ([^\n]+)($|\n.*)"
+
+    # re.match() returns an array with a tuple of (matched-string, match_1, ...)
+    m = re.match(match_re, commit_message)
+    if m and len(m) > 0:
+        os_only = m[0][2]
+        os_only_list = re.split(r'[, ]+', os_only)
+    else:
+        os_only_list = operating_systems
+
+    for os in operating_systems:
+        os_enabled = os in os_only_list
+        cenv['CI_{0}_ENABLED'.format(os.upper())] = os_enabled
+    ###
 
     return cenv
 
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 0cf6ea67ba5..744b1255ed9 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -72,7 +72,7 @@ task:
   # push-wait-for-ci cycle time a bit when debugging operating system specific
   # failures. Uses skip instead of only_if, as cirrus otherwise warns about
   # only_if conditions not matching.
-  skip: $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:.*'
+  skip: $CI_SANITYCHECK_ENABLED == false
 
   env:
     CPUS: 4
@@ -167,7 +167,7 @@ task:
   <<: *freebsd_task_template
 
   depends_on: SanityCheck
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*freebsd.*'
+  only_if: $CI_FREEBSD_ENABLED
 
   sysinfo_script: |
     id
@@ -258,7 +258,7 @@ task:
     - name: NetBSD - Meson
       # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
       trigger_type: $CI_TRIGGER_TYPE_NETBSD
-      only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*netbsd.*'
+      only_if: $CI_NETBSD_ENABLED
       env:
         OS_NAME: netbsd
         IMAGE_FAMILY: pg-ci-netbsd-postgres
@@ -277,7 +277,7 @@ task:
     - name: OpenBSD - Meson
       # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
       trigger_type: $CI_TRIGGER_TYPE_OPENBSD
-      only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*openbsd.*'
+      only_if: $CI_OPENBSD_ENABLED
       env:
         OS_NAME: openbsd
         IMAGE_FAMILY: pg-ci-openbsd-postgres
@@ -417,7 +417,7 @@ task:
   <<: *linux_task_template
 
   depends_on: SanityCheck
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*'
+  only_if: $CI_LINUX_ENABLED
 
   ccache_cache:
     folder: ${CCACHE_DIR}
@@ -605,7 +605,7 @@ task:
   <<: *macos_task_template
 
   depends_on: SanityCheck
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*'
+  only_if: $CI_MACOS_ENABLED
 
   sysinfo_script: |
     id
@@ -711,7 +711,7 @@ task:
   <<: *windows_task_template
 
   depends_on: SanityCheck
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*'
+  only_if: $CI_WINDOWS_ENABLED
 
   setup_additional_packages_script: |
     REM choco install -y --no-progress ...
@@ -750,7 +750,7 @@ task:
   trigger_type: $CI_TRIGGER_TYPE_MINGW
 
   depends_on: SanityCheck
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*'
+  only_if: $CI_MINGW_ENABLED
 
   env:
     TEST_JOBS: 4 # higher concurrency causes occasional failures
@@ -804,10 +804,9 @@ task:
 
   # To limit unnecessary work only run this once the SanityCheck
   # succeeds. This is particularly important for this task as we intentionally
-  # use always: to continue after failures. Task that did not run count as a
-  # success, so we need to recheck SanityChecks's condition here ...
+  # use always: to continue after failures.
   depends_on: SanityCheck
-  only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*'
+  only_if: $CI_COMPILERWARNINGS_ENABLED
 
   env:
     CPUS: 4
-- 
2.49.0



^ permalink  raw  reply  [nested|flat] 30+ messages in thread

* Re: ci: Allow running mingw tests by default via environment variable
  2024-04-25 12:02 Re: ci: Allow running mingw tests by default via environment variable Nazir Bilal Yavuz <[email protected]>
  2025-03-04 00:27 ` Re: ci: Allow running mingw tests by default via environment variable Thomas Munro <[email protected]>
@ 2025-03-04 12:58   ` Andres Freund <[email protected]>
  1 sibling, 0 replies; 30+ messages in thread

From: Andres Freund @ 2025-03-04 12:58 UTC (permalink / raw)
  To: Thomas Munro <[email protected]>; +Cc: Nazir Bilal Yavuz <[email protected]>; pgsql-hackers; Wolfgang Walther <[email protected]>

Hi,

On 2025-03-04 13:27:29 +1300, Thomas Munro wrote:
> On Fri, Apr 26, 2024 at 12:02 AM Nazir Bilal Yavuz <[email protected]> wrote:
> > On Sat, 13 Apr 2024 at 05:12, Andres Freund <[email protected]> wrote:
> > > I propose that we instead run the task automatically if
> > > $REPO_MINGW_TRIGGER_BY_DEFAULT is set, typically in cirrus' per-repository
> > > configuration.
> > >
> > > Unfortunately that's somewhat awkward to do in the cirrus-ci yaml
> > > configuration, the set of conditional expressions supported is very
> > > simplistic.
> > >
> > > To deal with that, I extended .cirrus.star to compute the required environment
> > > variable. If $REPO_MINGW_TRIGGER_BY_DEFAULT is set, CI_MINGW_TRIGGER_TYPE is
> > > set to 'automatic', if not it's 'manual'.
> >
> > Changes look good to me. My only complaint could be using only 'true'
> > for the REPO_MINGW_TRIGGER_BY_DEFAULT, not a possible list of values
> > but this is not important.
> 
> Here's a generalised version of 0001.  If you put this in your
> repository settings on Cirrus's website:
> 
> REPO_CI_AUTOMATIC_TRIGGER_TASKS="mingw netbsd"
> 
> ... then it defines:
> 
> CI_TRIGGER_TYPE_MINGW=automatic
> CI_TRIGGER_TYPE_NETBSD=automatic
> CI_TRIGGER_TYPE_OPENBSD=manual

Nice!


> The set of tasks that get this treatment is defined by this line in
> .cirrus.star:
> 
>     default_manual_trigger_tasks = ['mingw', 'netbsd', 'openbsd']
> 
> Then the individual tasks in .cirrus.tasks.yml use those variables:
> 
>      - name: NetBSD - Meson
> +      # See REPO_CI_AUTOMATIC_TRIGGER_TASKS in .cirrus.star
> +      trigger_type: $CI_TRIGGER_TYPE_NETBSD
> 
> Unfortunately the funky Starlark language doesn't seem to have regular
> expressions, so it's just searching for those substrings without
> contemplating delimiters.  That doesn't seem to be a problem at this
> scale...

FWIW, It does:

https://cirrus-ci.org/guide/programming-tasks/#re
https://github.com/qri-io/starlib/tree/master/re


Your edited version is looking good to me!

Greetings,

Andres Freund






^ permalink  raw  reply  [nested|flat] 30+ messages in thread

end of thread, other threads:[~2025-04-07 13:03 UTC | newest]

Thread overview: 30+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2017-09-05 03:59 cache lookup errors for missing replication origins Michael Paquier <[email protected]>
2017-09-06 06:51 ` Michael Paquier <[email protected]>
2021-01-07 05:28 [PATCH 1/2] psql \dX: list extended statistics objects Tatsuro Yamada <[email protected]>
2021-01-07 05:28 [PATCH 1/2] psql \dX: list extended statistics objects Tatsuro Yamada <[email protected]>
2021-01-07 05:28 [PATCH 1/2] psql \dX: list extended statistics objects Tatsuro Yamada <[email protected]>
2021-01-07 05:28 [PATCH 1/2] psql \dX: list extended statistics objects Tatsuro Yamada <[email protected]>
2021-01-07 05:28 [PATCH 1/2] psql \dX: list extended statistics objects Tatsuro Yamada <[email protected]>
2021-01-07 05:28 [PATCH 1/2] psql \dX: list extended statistics objects Tatsuro Yamada <[email protected]>
2021-01-07 05:28 [PATCH 1/2] psql \dX: list extended statistics objects Tatsuro Yamada <[email protected]>
2021-01-07 05:28 [PATCH 1/2] psql \dX: list extended statistics objects Tatsuro Yamada <[email protected]>
2021-01-07 05:28 [PATCH 1/2] psql \dX: list extended statistics objects Tatsuro Yamada <[email protected]>
2021-01-07 05:28 [PATCH 1/2] psql \dX: list extended statistics objects Tatsuro Yamada <[email protected]>
2021-01-07 05:28 [PATCH 1/2] psql \dX: list extended statistics objects Tatsuro Yamada <[email protected]>
2021-01-07 05:28 [PATCH 1/2] psql \dX: list extended statistics objects Tatsuro Yamada <[email protected]>
2021-01-07 05:28 [PATCH 1/2] psql \dX: list extended statistics objects Tatsuro Yamada <[email protected]>
2021-01-07 05:28 [PATCH 1/2] psql \dX: list extended statistics objects Tatsuro Yamada <[email protected]>
2021-01-07 05:28 [PATCH 1/2] psql \dX: list extended statistics objects Tatsuro Yamada <[email protected]>
2021-01-07 05:28 [PATCH 1/2] psql \dX: list extended statistics objects Tatsuro Yamada <[email protected]>
2021-01-07 05:28 [PATCH 1/2] psql \dX: list extended statistics objects Tatsuro Yamada <[email protected]>
2021-01-07 05:28 [PATCH 1/2] psql \dX: list extended statistics objects Tatsuro Yamada <[email protected]>
2021-01-07 05:28 [PATCH 1/2] psql \dX: list extended statistics objects Tatsuro Yamada <[email protected]>
2021-01-07 05:28 [PATCH 1/2] psql \dX: list extended statistics objects Tatsuro Yamada <[email protected]>
2021-01-07 05:28 [PATCH 1/2] psql \dX: list extended statistics objects Tatsuro Yamada <[email protected]>
2024-04-25 12:02 Re: ci: Allow running mingw tests by default via environment variable Nazir Bilal Yavuz <[email protected]>
2025-03-04 00:27 ` Re: ci: Allow running mingw tests by default via environment variable Thomas Munro <[email protected]>
2025-03-04 08:53   ` Re: ci: Allow running mingw tests by default via environment variable Nazir Bilal Yavuz <[email protected]>
2025-03-04 13:00     ` Re: ci: Allow running mingw tests by default via environment variable Andres Freund <[email protected]>
2025-03-05 15:51       ` Re: ci: Allow running mingw tests by default via environment variable Andres Freund <[email protected]>
2025-04-07 13:03         ` Re: ci: Allow running mingw tests by default via environment variable Nazir Bilal Yavuz <[email protected]>
2025-03-04 12:58   ` Re: ci: Allow running mingw tests by default via environment variable Andres Freund <[email protected]>

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox