public inbox for [email protected]
help / color / mirror / Atom feed[PATCH v24 1/7] Built-in compression method
14+ messages / 5 participants
[nested] [flat]
* [PATCH v24 1/7] Built-in compression method
@ 2021-02-05 12:51 dilipkumar <[email protected]>
0 siblings, 0 replies; 14+ messages in thread
From: dilipkumar @ 2021-02-05 12:51 UTC (permalink / raw)
Add syntax allowing a compression method to be specified.
As of now there is only 2 option for build-in compression
method (pglz, lz4) which can be set while creating a table
or adding a new column. No option for altering the
compression method for an existing column.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Robert Haas and Tomas Vondra.
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
Discussions:
https://www.postgresql.org/message-id/[email protected]
https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40...
https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail...
https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development
https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail...
---
configure | 116 ++++++++++
configure.ac | 17 ++
contrib/test_decoding/expected/ddl.out | 50 ++---
doc/src/sgml/ddl.sgml | 3 +
doc/src/sgml/ref/create_table.sgml | 25 +++
src/backend/access/Makefile | 2 +-
src/backend/access/brin/brin_tuple.c | 5 +-
src/backend/access/common/detoast.c | 103 ++++++---
src/backend/access/common/indextuple.c | 4 +-
src/backend/access/common/toast_internals.c | 63 +++---
src/backend/access/common/tupdesc.c | 8 +
src/backend/access/compression/Makefile | 17 ++
src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++
.../access/compression/compress_pglz.c | 138 ++++++++++++
src/backend/access/table/toast_helper.c | 5 +-
src/backend/bootstrap/bootstrap.c | 5 +
src/backend/catalog/genbki.pl | 7 +-
src/backend/catalog/heap.c | 4 +
src/backend/catalog/index.c | 2 +
src/backend/catalog/toasting.c | 5 +
src/backend/commands/amcmds.c | 10 +
src/backend/commands/createas.c | 14 ++
src/backend/commands/matview.c | 14 ++
src/backend/commands/tablecmds.c | 111 +++++++++-
src/backend/executor/nodeModifyTable.c | 122 +++++++++++
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/nodes/nodeFuncs.c | 2 +
src/backend/nodes/outfuncs.c | 1 +
src/backend/parser/gram.y | 26 ++-
src/backend/parser/parse_utilcmd.c | 8 +
src/backend/utils/adt/pseudotypes.c | 1 +
src/backend/utils/adt/varlena.c | 42 ++++
src/bin/pg_dump/pg_backup.h | 1 +
src/bin/pg_dump/pg_dump.c | 36 +++-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/pg_dump/t/002_pg_dump.pl | 12 +-
src/bin/psql/describe.c | 28 ++-
src/include/access/compressamapi.h | 99 +++++++++
src/include/access/detoast.h | 8 +
src/include/access/toast_helper.h | 1 +
src/include/access/toast_internals.h | 19 +-
src/include/catalog/pg_am.dat | 6 +
src/include/catalog/pg_am.h | 1 +
src/include/catalog/pg_attribute.h | 8 +-
src/include/catalog/pg_proc.dat | 20 ++
src/include/catalog/pg_type.dat | 5 +
src/include/commands/defrem.h | 1 +
src/include/executor/executor.h | 4 +-
src/include/nodes/execnodes.h | 6 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 2 +
src/include/parser/kwlist.h | 1 +
src/include/pg_config.h.in | 3 +
src/include/postgres.h | 12 +-
src/test/regress/expected/compression.out | 201 ++++++++++++++++++
src/test/regress/expected/compression_1.out | 189 ++++++++++++++++
src/test/regress/expected/psql.out | 68 +++---
src/test/regress/parallel_schedule | 2 +-
src/test/regress/serial_schedule | 1 +
src/test/regress/sql/compression.sql | 85 ++++++++
src/tools/msvc/Solution.pm | 1 +
src/tools/pgindent/typedefs.list | 1 +
63 files changed, 1771 insertions(+), 148 deletions(-)
create mode 100644 src/backend/access/compression/Makefile
create mode 100644 src/backend/access/compression/compress_lz4.c
create mode 100644 src/backend/access/compression/compress_pglz.c
create mode 100644 src/include/access/compressamapi.h
create mode 100644 src/test/regress/expected/compression.out
create mode 100644 src/test/regress/expected/compression_1.out
create mode 100644 src/test/regress/sql/compression.sql
diff --git a/configure b/configure
index ce9ea36999..0895b2f326 100755
--- a/configure
+++ b/configure
@@ -699,6 +699,7 @@ with_gnu_ld
LD
LDFLAGS_SL
LDFLAGS_EX
+with_lz4
with_zlib
with_system_tzdata
with_libxslt
@@ -864,6 +865,7 @@ with_libxml
with_libxslt
with_system_tzdata
with_zlib
+with_lz4
with_gnu_ld
with_ssl
with_openssl
@@ -1569,6 +1571,7 @@ Optional Packages:
--with-system-tzdata=DIR
use system time zone data in DIR
--without-zlib do not use Zlib
+ --with-lz4 build with LZ4 support
--with-gnu-ld assume the C compiler uses GNU ld [default=no]
--with-ssl=LIB use LIB for SSL/TLS support (openssl)
--with-openssl obsolete spelling of --with-ssl=openssl
@@ -8563,6 +8566,39 @@ fi
+#
+# LZ4
+#
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5
+$as_echo_n "checking whether to build with LZ4 support... " >&6; }
+
+
+
+# Check whether --with-lz4 was given.
+if test "${with_lz4+set}" = set; then :
+ withval=$with_lz4;
+ case $withval in
+ yes)
+
+$as_echo "#define USE_LZ4 1" >>confdefs.h
+
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_lz4=no
+
+fi
+
+
+
+
#
# Assignments
#
@@ -12054,6 +12090,56 @@ fi
fi
+if test "$with_lz4" = yes ; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5
+$as_echo_n "checking for LZ4_compress in -llz4... " >&6; }
+if ${ac_cv_lib_lz4_LZ4_compress+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ ac_check_lib_save_LIBS=$LIBS
+LIBS="-llz4 $LIBS"
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+
+/* Override any GCC internal prototype to avoid an error.
+ Use char because int might match the return type of a GCC
+ builtin and then its argument prototype would still apply. */
+#ifdef __cplusplus
+extern "C"
+#endif
+char LZ4_compress ();
+int
+main ()
+{
+return LZ4_compress ();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ ac_cv_lib_lz4_LZ4_compress=yes
+else
+ ac_cv_lib_lz4_LZ4_compress=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5
+$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; }
+if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then :
+ cat >>confdefs.h <<_ACEOF
+#define HAVE_LIBLZ4 1
+_ACEOF
+
+ LIBS="-llz4 $LIBS"
+
+else
+ as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5
+fi
+
+fi
+
if test "$enable_spinlocks" = yes; then
$as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h
@@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5
fi
+fi
+
+if test "$with_lz4" = yes ; then
+ for ac_header in lz4/lz4.h
+do :
+ ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default"
+if test "x$ac_cv_header_lz4_lz4_h" = xyes; then :
+ cat >>confdefs.h <<_ACEOF
+#define HAVE_LZ4_LZ4_H 1
+_ACEOF
+
+else
+ for ac_header in lz4.h
+do :
+ ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default"
+if test "x$ac_cv_header_lz4_h" = xyes; then :
+ cat >>confdefs.h <<_ACEOF
+#define HAVE_LZ4_H 1
+_ACEOF
+
+else
+ as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5
+fi
+
+done
+
+fi
+
+done
+
fi
if test "$with_gssapi" = yes ; then
diff --git a/configure.ac b/configure.ac
index 07da84d401..63940b78a0 100644
--- a/configure.ac
+++ b/configure.ac
@@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes,
[do not use Zlib])
AC_SUBST(with_zlib)
+#
+# LZ4
+#
+AC_MSG_CHECKING([whether to build with LZ4 support])
+PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support],
+ [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])])
+AC_SUBST(with_lz4)
+
#
# Assignments
#
@@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory.
Use --without-zlib to disable zlib support.])])
fi
+if test "$with_lz4" = yes ; then
+ AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])])
+fi
+
if test "$enable_spinlocks" = yes; then
AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.])
else
@@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory.
Use --without-zlib to disable zlib support.])])
fi
+if test "$with_lz4" = yes ; then
+ AC_CHECK_HEADERS(lz4/lz4.h, [],
+ [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])])
+fi
+
if test "$with_gssapi" = yes ; then
AC_CHECK_HEADERS(gssapi/gssapi.h, [],
[AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])])
diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out
index 4ff0044c78..6ee077678d 100644
--- a/contrib/test_decoding/expected/ddl.out
+++ b/contrib/test_decoding/expected/ddl.out
@@ -438,12 +438,12 @@ CREATE TABLE replication_metadata (
WITH (user_catalog_table = true)
;
\d+ replication_metadata
- Table "public.replication_metadata"
- Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
-----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+-------------
- id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | |
- relation | name | | not null | | plain | |
- options | text[] | | | | extended | |
+ Table "public.replication_metadata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+-------------
+ id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | |
+ relation | name | | not null | | plain | | |
+ options | text[] | | | | extended | pglz | |
Indexes:
"replication_metadata_pkey" PRIMARY KEY, btree (id)
Options: user_catalog_table=true
@@ -452,12 +452,12 @@ INSERT INTO replication_metadata(relation, options)
VALUES ('foo', ARRAY['a', 'b']);
ALTER TABLE replication_metadata RESET (user_catalog_table);
\d+ replication_metadata
- Table "public.replication_metadata"
- Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
-----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+-------------
- id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | |
- relation | name | | not null | | plain | |
- options | text[] | | | | extended | |
+ Table "public.replication_metadata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+-------------
+ id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | |
+ relation | name | | not null | | plain | | |
+ options | text[] | | | | extended | pglz | |
Indexes:
"replication_metadata_pkey" PRIMARY KEY, btree (id)
@@ -465,12 +465,12 @@ INSERT INTO replication_metadata(relation, options)
VALUES ('bar', ARRAY['a', 'b']);
ALTER TABLE replication_metadata SET (user_catalog_table = true);
\d+ replication_metadata
- Table "public.replication_metadata"
- Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
-----------+---------+-----------+----------+--------------------------------------------------+----------+--------------+-------------
- id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | |
- relation | name | | not null | | plain | |
- options | text[] | | | | extended | |
+ Table "public.replication_metadata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+----------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+-------------
+ id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | |
+ relation | name | | not null | | plain | | |
+ options | text[] | | | | extended | pglz | |
Indexes:
"replication_metadata_pkey" PRIMARY KEY, btree (id)
Options: user_catalog_table=true
@@ -483,13 +483,13 @@ ALTER TABLE replication_metadata ALTER COLUMN rewritemeornot TYPE text;
ERROR: cannot rewrite table "replication_metadata" used as a catalog table
ALTER TABLE replication_metadata SET (user_catalog_table = false);
\d+ replication_metadata
- Table "public.replication_metadata"
- Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
-----------------+---------+-----------+----------+--------------------------------------------------+----------+--------------+-------------
- id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | |
- relation | name | | not null | | plain | |
- options | text[] | | | | extended | |
- rewritemeornot | integer | | | | plain | |
+ Table "public.replication_metadata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+----------------+---------+-----------+----------+--------------------------------------------------+----------+-------------+--------------+-------------
+ id | integer | | not null | nextval('replication_metadata_id_seq'::regclass) | plain | | |
+ relation | name | | not null | | plain | | |
+ options | text[] | | | | extended | pglz | |
+ rewritemeornot | integer | | | | plain | | |
Indexes:
"replication_metadata_pkey" PRIMARY KEY, btree (id)
Options: user_catalog_table=false
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 1e9a4625cc..4d7ed698b9 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -3762,6 +3762,9 @@ CREATE TABLE measurement (
<productname>PostgreSQL</productname>
tables (or, possibly, foreign tables). It is possible to specify a
tablespace and storage parameters for each partition separately.
+ By default, each column in a partition inherits the compression method
+ from parent table's column, however a different compression method can be
+ set for each partition.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index 569f4c9da7..51a7a977a5 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI
GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] |
UNIQUE <replaceable class="parameter">index_parameters</replaceable> |
PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> |
+ COMPRESSION <replaceable class="parameter">compression_method</replaceable> |
REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ]
[ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] }
[ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>INCLUDING COMPRESSION</literal></term>
+ <listitem>
+ <para>
+ Compression method of the columns will be copied. The default
+ behavior is to exclude compression methods, resulting in the columns
+ having the default compression method.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>INCLUDING CONSTRAINTS</literal></term>
<listitem>
@@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term>
+ <listitem>
+ <para>
+ This sets the compression method for a column. The supported compression
+ methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ <literal>lz4</literal> is available only if <literal>--with-lz4</literal>
+ was used when building <productname>PostgreSQL</productname>. The default
+ is <literal>pglz</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="sql-createtable-exclude">
<term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term>
<listitem>
diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile
index 0880e0a8bb..ba08bdd63e 100644
--- a/src/backend/access/Makefile
+++ b/src/backend/access/Makefile
@@ -8,7 +8,7 @@ subdir = src/backend/access
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \
+SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \
table tablesample transam
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c
index a7eb1c9473..0ab5712c71 100644
--- a/src/backend/access/brin/brin_tuple.c
+++ b/src/backend/access/brin/brin_tuple.c
@@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple,
(atttype->typstorage == TYPSTORAGE_EXTENDED ||
atttype->typstorage == TYPSTORAGE_MAIN))
{
- Datum cvalue = toast_compress_datum(value);
+ Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc,
+ keyno);
+ Datum cvalue = toast_compress_datum(value,
+ att->attcompression);
if (DatumGetPointer(cvalue) != NULL)
{
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index d1cdbaf648..b78d49167b 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -13,6 +13,7 @@
#include "postgres.h"
+#include "access/compressamapi.h"
#include "access/detoast.h"
#include "access/table.h"
#include "access/tableam.h"
@@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
}
/* ----------
- * toast_decompress_datum -
+ * toast_get_compression_oid -
*
- * Decompress a compressed version of a varlena datum
+ * Returns the Oid of the compression method stored in the compressed data. If
+ * the varlena is not compressed then returns InvalidOid.
*/
-static struct varlena *
-toast_decompress_datum(struct varlena *attr)
+Oid
+toast_get_compression_oid(struct varlena *attr)
{
- struct varlena *result;
+ if (VARATT_IS_EXTERNAL_ONDISK(attr))
+ {
+ struct varatt_external toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+
+ /* fast path for non-compressed external datums */
+ if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer))
+ return InvalidOid;
+
+ /*
+ * Just fetch the toast compress header to know the compression method
+ * in the compressed data.
+ */
+ attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS);
+ }
+ else if (!VARATT_IS_COMPRESSED(attr))
+ return InvalidOid;
+
+ return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+}
+
+/* ----------
+ * toast_get_compression_handler - get the compression handler routines
+ *
+ * helper function for toast_decompress_datum and toast_decompress_datum_slice
+ */
+static inline const CompressionAmRoutine *
+toast_get_compression_handler(struct varlena *attr)
+{
+ const CompressionAmRoutine *cmroutine;
+ CompressionId cmid;
Assert(VARATT_IS_COMPRESSED(attr));
- result = (struct varlena *)
- palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ);
- SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ);
+ cmid = TOAST_COMPRESS_METHOD(attr);
- if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr),
- TOAST_COMPRESS_SIZE(attr),
- VARDATA(result),
- TOAST_COMPRESS_RAWSIZE(attr), true) < 0)
- elog(ERROR, "compressed data is corrupted");
+ /* Get the handler routines for the compression method */
+ switch (cmid)
+ {
+ case PGLZ_COMPRESSION_ID:
+ cmroutine = &pglz_compress_methods;
+ break;
+ case LZ4_COMPRESSION_ID:
+ cmroutine = &lz4_compress_methods;
+ break;
+ default:
+ elog(ERROR, "invalid compression method id %d", cmid);
+ }
- return result;
+ return cmroutine;
+}
+
+/* ----------
+ * toast_decompress_datum -
+ *
+ * Decompress a compressed version of a varlena datum
+ */
+static struct varlena *
+toast_decompress_datum(struct varlena *attr)
+{
+ const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+
+ return cmroutine->datum_decompress(attr);
}
@@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- struct varlena *result;
- int32 rawsize;
-
- Assert(VARATT_IS_COMPRESSED(attr));
-
- result = (struct varlena *) palloc(slicelength + VARHDRSZ);
-
- rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr),
- VARSIZE(attr) - TOAST_COMPRESS_HDRSZ,
- VARDATA(result),
- slicelength, false);
- if (rawsize < 0)
- elog(ERROR, "compressed data is corrupted");
+ const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
- SET_VARSIZE(result, rawsize + VARHDRSZ);
- return result;
+ /*
+ * If the handler supports the slice decompression then decompress the
+ * slice otherwise decompress complete data.
+ */
+ if (cmroutine->datum_decompress_slice)
+ return cmroutine->datum_decompress_slice(attr, slicelength);
+ else
+ return cmroutine->datum_decompress(attr);
}
/* ----------
diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c
index b72a138497..1d43d5d2ff 100644
--- a/src/backend/access/common/indextuple.c
+++ b/src/backend/access/common/indextuple.c
@@ -16,6 +16,7 @@
#include "postgres.h"
+#include "access/compressamapi.h"
#include "access/detoast.h"
#include "access/heaptoast.h"
#include "access/htup_details.h"
@@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor,
(att->attstorage == TYPSTORAGE_EXTENDED ||
att->attstorage == TYPSTORAGE_MAIN))
{
- Datum cvalue = toast_compress_datum(untoasted_values[i]);
+ Datum cvalue = toast_compress_datum(untoasted_values[i],
+ att->attcompression);
if (DatumGetPointer(cvalue) != NULL)
{
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index 9b9da0f41b..b04c5a5eb8 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid);
* ----------
*/
Datum
-toast_compress_datum(Datum value)
+toast_compress_datum(Datum value, Oid cmoid)
{
- struct varlena *tmp;
- int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value));
- int32 len;
+ struct varlena *tmp = NULL;
+ int32 valsize;
+ const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value)));
- /*
- * No point in wasting a palloc cycle if value size is out of the allowed
- * range for compression
- */
- if (valsize < PGLZ_strategy_default->min_input_size ||
- valsize > PGLZ_strategy_default->max_input_size)
- return PointerGetDatum(NULL);
+ Assert(OidIsValid(cmoid));
+
+ /* Get the handler routines for the compression method */
+ switch (cmoid)
+ {
+ case PGLZ_COMPRESSION_AM_OID:
+ cmroutine = &pglz_compress_methods;
+ break;
+ case LZ4_COMPRESSION_AM_OID:
+ cmroutine = &lz4_compress_methods;
+ break;
+ default:
+ elog(ERROR, "Invalid compression method oid %u", cmoid);
+ }
- tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- TOAST_COMPRESS_HDRSZ);
+ /* Call the actual compression function */
+ tmp = cmroutine->datum_compress((const struct varlena *) value);
+ if (!tmp)
+ return PointerGetDatum(NULL);
/*
- * We recheck the actual size even if pglz_compress() reports success,
- * because it might be satisfied with having saved as little as one byte
- * in the compressed data --- which could turn into a net loss once you
- * consider header and alignment padding. Worst case, the compressed
- * format might require three padding bytes (plus header, which is
- * included in VARSIZE(tmp)), whereas the uncompressed format would take
- * only one header byte and no padding if the value is short enough. So
- * we insist on a savings of more than 2 bytes to ensure we have a gain.
+ * We recheck the actual size even if compression reports success, because
+ * it might be satisfied with having saved as little as one byte in the
+ * compressed data --- which could turn into a net loss once you consider
+ * header and alignment padding. Worst case, the compressed format might
+ * require three padding bytes (plus header, which is included in
+ * VARSIZE(tmp)), whereas the uncompressed format would take only one
+ * header byte and no padding if the value is short enough. So we insist
+ * on a savings of more than 2 bytes to ensure we have a gain.
*/
- len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)),
- valsize,
- TOAST_COMPRESS_RAWDATA(tmp),
- PGLZ_strategy_default);
- if (len >= 0 &&
- len + TOAST_COMPRESS_HDRSZ < valsize - 2)
+ valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value));
+
+ if (VARSIZE(tmp) < valsize - 2)
{
- TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize);
- SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ);
/* successful compression */
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c
index 902f59440c..ca26fab487 100644
--- a/src/backend/access/common/tupdesc.c
+++ b/src/backend/access/common/tupdesc.c
@@ -19,6 +19,7 @@
#include "postgres.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "access/tupdesc_details.h"
#include "catalog/pg_collation.h"
@@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2)
return false;
if (attr1->attcollation != attr2->attcollation)
return false;
+ if (attr1->attcompression != attr2->attcompression)
+ return false;
/* attacl, attoptions and attfdwoptions are not even present... */
}
@@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc,
att->attstorage = typeForm->typstorage;
att->attcollation = typeForm->typcollation;
+ if (IsStorageCompressible(typeForm->typstorage))
+ att->attcompression = DefaultCompressionOid;
+ else
+ att->attcompression = InvalidOid;
+
ReleaseSysCache(tuple);
}
diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile
new file mode 100644
index 0000000000..779f54e785
--- /dev/null
+++ b/src/backend/access/compression/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile--
+# Makefile for access/compression
+#
+# IDENTIFICATION
+# src/backend/access/compression/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/backend/access/compression
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+
+OBJS = compress_pglz.o compress_lz4.o
+
+include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
new file mode 100644
index 0000000000..1856cf7df7
--- /dev/null
+++ b/src/backend/access/compression/compress_lz4.c
@@ -0,0 +1,164 @@
+/*-------------------------------------------------------------------------
+ *
+ * compress_lz4.c
+ * lz4 compression method.
+ *
+ * Portions Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/access/compression/compress_lz4.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#ifdef HAVE_LIBLZ4
+#include <lz4.h>
+#endif
+
+#include "access/compressamapi.h"
+#include "fmgr.h"
+#include "utils/builtins.h"
+
+/*
+ * lz4_cmcompress - compression routine for lz4 compression method
+ *
+ * Compresses source into dest using the default strategy. Returns the
+ * compressed varlena, or NULL if compression fails.
+ */
+static struct varlena *
+lz4_cmcompress(const struct varlena *value)
+{
+#ifndef HAVE_LIBLZ4
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#else
+ int32 valsize;
+ int32 len;
+ int32 max_size;
+ struct varlena *tmp = NULL;
+
+ valsize = VARSIZE_ANY_EXHDR(value);
+
+ /*
+ * Get maximum size of the compressed data that lz4 compression may output
+ * and allocate the memory for holding the compressed data and the header.
+ */
+ max_size = LZ4_compressBound(valsize);
+ tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+
+ len = LZ4_compress_default(VARDATA_ANY(value),
+ (char *) tmp + VARHDRSZ_COMPRESS,
+ valsize, max_size);
+ if (len <= 0)
+ elog(ERROR, "could not compress data with lz4");
+
+ /* data is incompressible so just free the memory and return NULL */
+ if (len > valsize)
+ {
+ pfree(tmp);
+ return NULL;
+ }
+
+ SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+
+ return tmp;
+#endif
+}
+
+/*
+ * lz4_cmdecompress - decompression routine for lz4 compression method
+ *
+ * Returns the decompressed varlena.
+ */
+static struct varlena *
+lz4_cmdecompress(const struct varlena *value)
+{
+#ifndef HAVE_LIBLZ4
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#else
+ int32 rawsize;
+ struct varlena *result;
+
+ /* allocate memory for holding the uncompressed data */
+ result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
+
+ /* decompress data using lz4 routine */
+ rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ VARDATA(result),
+ VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARRAWSIZE_4B_C(value));
+ if (rawsize < 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg_internal("compressed lz4 data is corrupt")));
+
+
+ SET_VARSIZE(result, rawsize + VARHDRSZ);
+
+ return result;
+#endif
+}
+
+/*
+ * lz4_cmdecompress_slice - slice decompression routine for lz4 compression
+ *
+ * Decompresses part of the data. Returns the decompressed varlena.
+ */
+static struct varlena *
+lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+{
+#ifndef HAVE_LIBLZ4
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#else
+ int32 rawsize;
+ struct varlena *result;
+
+ /* allocate memory for holding the uncompressed data */
+ result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
+
+ /* decompress partial data using lz4 routine */
+ rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ VARDATA(result),
+ VARSIZE(value) - VARHDRSZ_COMPRESS,
+ slicelength,
+ VARRAWSIZE_4B_C(value));
+ if (rawsize < 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg_internal("compressed lz4 data is corrupt")));
+
+ SET_VARSIZE(result, rawsize + VARHDRSZ);
+
+ return result;
+#endif
+}
+
+/* ------------------------------------------------------------------------
+ * Definition of the lz4 compression access method.
+ * ------------------------------------------------------------------------
+ */
+const CompressionAmRoutine lz4_compress_methods = {
+ .type = T_CompressionAmRoutine,
+ .datum_compress = lz4_cmcompress,
+ .datum_decompress = lz4_cmdecompress,
+ .datum_decompress_slice = lz4_cmdecompress_slice
+};
+
+/* lz4 compression handler function */
+Datum
+lz4handler(PG_FUNCTION_ARGS)
+{
+#ifndef HAVE_LIBLZ4
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#else
+ PG_RETURN_POINTER(&lz4_compress_methods);
+#endif
+}
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
new file mode 100644
index 0000000000..8a4bf427cf
--- /dev/null
+++ b/src/backend/access/compression/compress_pglz.c
@@ -0,0 +1,138 @@
+/*-------------------------------------------------------------------------
+ *
+ * compress_pglz.c
+ * pglz compression method
+ *
+ * Portions Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/access/compression/compress_pglz.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "common/pg_lzcompress.h"
+
+#include "fmgr.h"
+#include "utils/builtins.h"
+
+/*
+ * pglz_cmcompress - compression routine for pglz compression method
+ *
+ * Compresses source into dest using the default strategy. Returns the
+ * compressed varlena, or NULL if compression fails.
+ */
+static struct varlena *
+pglz_cmcompress(const struct varlena *value)
+{
+ int32 valsize,
+ len;
+ struct varlena *tmp = NULL;
+
+ valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value));
+
+ /*
+ * No point in wasting a palloc cycle if value size is outside the allowed
+ * range for compression.
+ */
+ if (valsize < PGLZ_strategy_default->min_input_size ||
+ valsize > PGLZ_strategy_default->max_input_size)
+ return NULL;
+
+ /*
+ * Get maximum size of the compressed data that pglz compression may output
+ * and allocate the memory for holding the compressed data and the header.
+ */
+ tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
+ VARHDRSZ_COMPRESS);
+
+ len = pglz_compress(VARDATA_ANY(value),
+ valsize,
+ (char *) tmp + VARHDRSZ_COMPRESS,
+ NULL);
+ if (len < 0)
+ {
+ pfree(tmp);
+ return NULL;
+ }
+
+ SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+
+ return tmp;
+}
+
+/*
+ * pglz_cmdecompress - decompression routine for pglz compression method
+ *
+ * Returns the decompressed varlena.
+ */
+static struct varlena *
+pglz_cmdecompress(const struct varlena *value)
+{
+ struct varlena *result;
+ int32 rawsize;
+
+ result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
+
+ rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
+ VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARDATA(result),
+ VARRAWSIZE_4B_C(value), true);
+ if (rawsize < 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg_internal("compressed pglz data is corrupt")));
+
+ SET_VARSIZE(result, rawsize + VARHDRSZ);
+
+ return result;
+}
+
+/*
+ * pglz_decompress - slice decompression routine for pglz compression method
+ *
+ * Decompresses part of the data. Returns the decompressed varlena.
+ */
+static struct varlena *
+pglz_cmdecompress_slice(const struct varlena *value,
+ int32 slicelength)
+{
+ struct varlena *result;
+ int32 rawsize;
+
+ result = (struct varlena *) palloc(slicelength + VARHDRSZ);
+
+ rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
+ VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARDATA(result),
+ slicelength, false);
+ if (rawsize < 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg_internal("compressed pglz data is corrupt")));
+
+ SET_VARSIZE(result, rawsize + VARHDRSZ);
+
+ return result;
+}
+
+/* ------------------------------------------------------------------------
+ * Definition of the pglz compression access method.
+ * ------------------------------------------------------------------------
+ */
+const CompressionAmRoutine pglz_compress_methods = {
+ .type = T_CompressionAmRoutine,
+ .datum_compress = pglz_cmcompress,
+ .datum_decompress = pglz_cmdecompress,
+ .datum_decompress_slice = pglz_cmdecompress_slice
+};
+
+/* pglz compression handler function */
+Datum
+pglzhandler(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_POINTER(&pglz_compress_methods);
+}
diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c
index fb36151ce5..53f78f9c3e 100644
--- a/src/backend/access/table/toast_helper.c
+++ b/src/backend/access/table/toast_helper.c
@@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc)
ttc->ttc_attr[i].tai_colflags = 0;
ttc->ttc_attr[i].tai_oldexternal = NULL;
+ ttc->ttc_attr[i].tai_compression = att->attcompression;
if (ttc->ttc_oldvalues != NULL)
{
@@ -226,9 +227,11 @@ void
toast_tuple_try_compression(ToastTupleContext *ttc, int attribute)
{
Datum *value = &ttc->ttc_values[attribute];
- Datum new_value = toast_compress_datum(*value);
+ Datum new_value;
ToastAttrInfo *attr = &ttc->ttc_attr[attribute];
+ new_value = toast_compress_datum(*value, attr->tai_compression);
+
if (DatumGetPointer(new_value) != NULL)
{
/* successful compression */
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..9b451eaa71 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -17,6 +17,7 @@
#include <unistd.h>
#include <signal.h>
+#include "access/compressamapi.h"
#include "access/genam.h"
#include "access/heapam.h"
#include "access/htup_details.h"
@@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness)
attrtypes[attnum]->attcacheoff = -1;
attrtypes[attnum]->atttypmod = -1;
attrtypes[attnum]->attislocal = true;
+ if (IsStorageCompressible(attrtypes[attnum]->attstorage))
+ attrtypes[attnum]->attcompression = DefaultCompressionOid;
+ else
+ attrtypes[attnum]->attcompression = InvalidOid;
if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL)
{
diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl
index b159958112..f5bb37b972 100644
--- a/src/backend/catalog/genbki.pl
+++ b/src/backend/catalog/genbki.pl
@@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId;
my $C_COLLATION_OID =
Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation},
'C_COLLATION_OID');
-
+my $PGLZ_COMPRESSION_AM_OID =
+ Catalog::FindDefinedSymbolFromData($catalog_data{pg_am},
+ 'PGLZ_COMPRESSION_AM_OID');
# Fill in pg_class.relnatts by looking at the referenced catalog's schema.
# This is ugly but there's no better place; Catalog::AddDefaultValues
@@ -906,6 +908,9 @@ sub morph_row_for_pgattr
$row->{attcollation} =
$type->{typcollation} ne '0' ? $C_COLLATION_OID : 0;
+ $row->{attcompression} =
+ $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0;
+
if (defined $attr->{forcenotnull})
{
$row->{attnotnull} = 't';
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index 9abc4a1f55..b53b6b50e6 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -29,6 +29,7 @@
*/
#include "postgres.h"
+#include "access/compressamapi.h"
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/multixact.h"
@@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel,
slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal);
slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount);
slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation);
+ slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression);
if (attoptions && attoptions[natts] != (Datum) 0)
slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts];
else
@@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum)
/* Unset this so no one tries to look up the generation expression */
attStruct->attgenerated = '\0';
+ attStruct->attcompression = InvalidOid;
+
/*
* Change the column name to something that isn't likely to conflict
*/
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 1514937748..d1d6bdf470 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -24,6 +24,7 @@
#include <unistd.h>
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/heapam.h"
#include "access/multixact.h"
#include "access/reloptions.h"
@@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation,
to->attbyval = from->attbyval;
to->attstorage = from->attstorage;
to->attalign = from->attalign;
+ to->attcompression = from->attcompression;
}
else
{
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c
index d7b806020d..a549481557 100644
--- a/src/backend/catalog/toasting.c
+++ b/src/backend/catalog/toasting.c
@@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN;
TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN;
+ /* Toast field should not be compressed */
+ TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid;
+ TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid;
+ TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid;
+
/*
* Toast tables for regular relations go in pg_toast; those for temp
* relations go into the per-backend temp-toast-table namespace.
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index eff9535ed0..3ad4a61739 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok)
return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok);
}
+/*
+ * get_compression_am_oid - given an access method name, look up its OID
+ * and verify it corresponds to an compression AM.
+ */
+Oid
+get_compression_am_oid(const char *amname, bool missing_ok)
+{
+ return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok);
+}
+
/*
* get_am_oid - given an access method name, look up its OID.
* The type is not checked.
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index dce882012e..1d17dc0d6b 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -61,6 +61,7 @@ typedef struct
CommandId output_cid; /* cmin to insert in output tuples */
int ti_options; /* table_tuple_insert performance options */
BulkInsertState bistate; /* bulk insert state */
+ TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */
} DR_intorel;
/* utility functions for CTAS definition creation */
@@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self)
/* Nothing to insert if WITH NO DATA is specified. */
if (!myState->into->skipData)
{
+ /*
+ * Compare the compression method of the compressed attribute in the
+ * source tuple with target attribute and if those are different then
+ * decompress those attributes.
+ */
+ slot = CompareCompressionMethodAndDecompress(slot,
+ &myState->decompress_tuple_slot,
+ myState->rel->rd_att);
+
/*
* Note that the input slot might not be of the type of the target
* relation. That's supported by table_tuple_insert(), but slightly
@@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self)
/* close rel, but keep lock until commit */
table_close(myState->rel, NoLock);
myState->rel = NULL;
+
+ /* release the slot used for decompressing the tuple */
+ if (myState->decompress_tuple_slot)
+ ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot);
}
/*
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index c5c25ce11d..713fc3fceb 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -56,6 +56,7 @@ typedef struct
CommandId output_cid; /* cmin to insert in output tuples */
int ti_options; /* table_tuple_insert performance options */
BulkInsertState bistate; /* bulk insert state */
+ TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */
} DR_transientrel;
static int matview_maintenance_depth = 0;
@@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self)
{
DR_transientrel *myState = (DR_transientrel *) self;
+ /*
+ * Compare the compression method of the compressed attribute in the
+ * source tuple with target attribute and if those are different then
+ * decompress those attributes.
+ */
+ slot = CompareCompressionMethodAndDecompress(slot,
+ &myState->decompress_tuple_slot,
+ myState->transientrel->rd_att);
+
/*
* Note that the input slot might not be of the type of the target
* relation. That's supported by table_tuple_insert(), but slightly less
@@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self)
/* close transientrel, but keep lock until commit */
table_close(myState->transientrel, NoLock);
myState->transientrel = NULL;
+
+ /* release the slot used for decompressing the tuple */
+ if (myState->decompress_tuple_slot)
+ ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot);
}
/*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 420991e315..dd81d5bf4e 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -15,6 +15,7 @@
#include "postgres.h"
#include "access/attmap.h"
+#include "access/compressamapi.h"
#include "access/genam.h"
#include "access/heapam.h"
#include "access/heapam_xlog.h"
@@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-
+static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
if (colDef->generated)
attr->attgenerated = colDef->generated;
+
+ /*
+ * lookup attribute's compression method and store its Oid in the
+ * attr->attcompression.
+ */
+ if (relkind == RELKIND_RELATION ||
+ relkind == RELKIND_PARTITIONED_TABLE ||
+ relkind == RELKIND_MATVIEW)
+ attr->attcompression =
+ GetAttributeCompression(attr, colDef->compression);
+ else
+ attr->attcompression = InvalidOid;
}
/*
@@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
storage_name(def->storage),
storage_name(attribute->attstorage))));
+ /* Copy/check compression parameter */
+ if (OidIsValid(attribute->attcompression))
+ {
+ char *compression = get_am_name(attribute->attcompression);
+
+ if (!def->compression)
+ def->compression = compression;
+ else if (strcmp(def->compression, compression) != 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("column \"%s\" has a compression method conflict",
+ attributeName),
+ errdetail("%s versus %s", def->compression, compression)));
+ }
+
def->inhcount++;
/* Merge of NOT NULL constraints = OR 'em together */
def->is_not_null |= attribute->attnotnull;
@@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
+ def->compression = get_am_name(attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
storage_name(def->storage),
storage_name(newdef->storage))));
+ /* Copy compression parameter */
+ if (!def->compression)
+ def->compression = newdef->compression;
+ else if (newdef->compression)
+ {
+ if (strcmp(def->compression, newdef->compression))
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("column \"%s\" has a compression method conflict",
+ attributeName),
+ errdetail("%s versus %s", def->compression, newdef->compression)));
+ }
+
/* Mark the column as locally defined */
def->is_local = true;
/* Merge of NOT NULL constraints = OR 'em together */
@@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
attribute.attislocal = colDef->is_local;
attribute.attinhcount = colDef->inhcount;
attribute.attcollation = collOid;
+
+ /*
+ * lookup attribute's compression method and store its Oid in the
+ * attr->attcompression.
+ */
+ if (rel->rd_rel->relkind == RELKIND_RELATION ||
+ rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
+ attribute.attcompression = GetAttributeCompression(&attribute,
+ colDef->compression);
+ else
+ attribute.attcompression = InvalidOid;
+
/* attribute.attacl is handled by InsertPgAttributeTuples() */
ReleaseSysCache(typeTuple);
@@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
ReleaseSysCache(typeTuple);
+ /* Setup attribute compression */
+ if (rel->rd_rel->relkind == RELKIND_RELATION ||
+ rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
+ {
+ /*
+ * InvalidOid for the plain/external storage otherwise default
+ * compression id.
+ */
+ if (!IsStorageCompressible(tform->typstorage))
+ attTup->attcompression = InvalidOid;
+ else if (!OidIsValid(attTup->attcompression))
+ attTup->attcompression = DefaultCompressionOid;
+ }
+ else
+ attTup->attcompression = InvalidOid;
+
CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup);
table_close(attrelation, RowExclusiveLock);
@@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
+
+/*
+ * resolve column compression specification to an OID.
+ */
+static Oid
+GetAttributeCompression(Form_pg_attribute att, char *compression)
+{
+ char typstorage = get_typstorage(att->atttypid);
+ Oid amoid;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return DefaultCompressionOid;
+
+ amoid = get_compression_am_oid(compression, false);
+
+#ifndef HAVE_LIBLZ4
+ if (amoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+ return amoid;
+}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index 2993ba43e3..f77deee399 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -37,6 +37,8 @@
#include "postgres.h"
+#include "access/compressamapi.h"
+#include "access/detoast.h"
#include "access/heapam.h"
#include "access/htup_details.h"
#include "access/tableam.h"
@@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate,
return slot;
}
+/*
+ * Compare the compression method of each compressed value with the
+ * compression method of the target attribute. If the compression method
+ * of the compressed value is not supported in the target attribute then
+ * decompress the value. If any of the value need to decompress then we
+ * need to store that into the new slot.
+ *
+ * The slot will hold the input slot but if any of the value need to be
+ * decompressed then the new slot will be stored into this and the old
+ * slot will be dropped.
+ */
+TupleTableSlot *
+CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
+ TupleTableSlot **outslot,
+ TupleDesc targetTupDesc)
+{
+ int i;
+ int attnum;
+ int natts = slot->tts_tupleDescriptor->natts;
+ bool decompressed_any = false;
+ bool slot_tup_deformed = false;
+ Oid cmoid;
+ TupleDesc tupleDesc = slot->tts_tupleDescriptor;
+
+ if (natts == 0)
+ return slot;
+
+ /*
+ * Loop over all the attributes in the tuple and check if any attribute is
+ * compressed and its compression method is not same as the target
+ * atrribute's compression method then decompress it.
+ */
+ for (i = 0; i < natts; i++)
+ {
+ attnum = tupleDesc->attrs[i].attnum;
+
+ if (TupleDescAttr(tupleDesc, i)->attlen == -1)
+ {
+ struct varlena *new_value;
+
+ /* fetch the values of all the attribute, if not already done */
+ if (!slot_tup_deformed)
+ {
+ slot_getallattrs(slot);
+ slot_tup_deformed = true;
+ }
+
+ /* nothing to be done, if the value is null */
+ if (slot->tts_isnull[attnum - 1])
+ continue;
+
+ new_value = (struct varlena *)
+ DatumGetPointer(slot->tts_values[attnum - 1]);
+
+ /*
+ * Get the compression method Oid stored in the toast header and
+ * compare it with the compression method of the target.
+ */
+ cmoid = toast_get_compression_oid(new_value);
+ if (OidIsValid(cmoid) &&
+ targetTupDesc->attrs[i].attcompression != cmoid)
+ {
+ new_value = detoast_attr(new_value);
+ slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
+ decompressed_any = true;
+ }
+ }
+ }
+
+ /*
+ * If we have decompressed any of the fields then we need to copy the
+ * values/null array from oldslot to the new slot and materialize the new
+ * slot.
+ */
+ if (decompressed_any)
+ {
+ TupleTableSlot *newslot = *outslot;
+
+ /*
+ * If the called has passed an invalid slot then create a new slot.
+ * Otherwise, just clear the existing tuple from the slot. This slot
+ * should be stored by the caller so that it can be reused for
+ * decompressing the subsequent tuples.
+ */
+ if (newslot == NULL)
+ newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops);
+ else
+ ExecClearTuple(newslot);
+
+ /*
+ * Copy the value/null array to the new slot and materialize it,
+ * before clearing the tuple from the old slot.
+ */
+ memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum));
+ memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool));
+ ExecStoreVirtualTuple(newslot);
+ ExecMaterializeSlot(newslot);
+ ExecClearTuple(slot);
+
+ *outslot = newslot;
+
+ return newslot;
+ }
+
+ return slot;
+}
+
/* ----------------------------------------------------------------
* ExecModifyTable
*
@@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate)
slot = ExecFilterJunk(junkfilter, slot);
}
+ /*
+ * Compare the compression method of the compressed attribute in the
+ * source tuple with target attribute and if those are different then
+ * decompress those attributes.
+ */
+ slot = CompareCompressionMethodAndDecompress(slot,
+ &node->mt_decompress_tuple_slot,
+ resultRelInfo->ri_RelationDesc->rd_att);
+
switch (operation)
{
case CMD_INSERT:
@@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node)
ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot);
}
+ /* release the slot used for decompressing the tuple */
+ if (node->mt_decompress_tuple_slot)
+ ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot);
+
/*
* Free the exprcontext
*/
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 65bbc18ecb..1338e04409 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
+ COPY_STRING_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index c2d73626fc..f3592003da 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
+ COMPARE_STRING_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c
index 49357ac5c2..38226530c6 100644
--- a/src/backend/nodes/nodeFuncs.c
+++ b/src/backend/nodes/nodeFuncs.c
@@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node,
if (walker(coldef->typeName, context))
return true;
+ if (walker(coldef->compression, context))
+ return true;
if (walker(coldef->raw_default, context))
return true;
if (walker(coldef->collClause, context))
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index f5dcedf6e8..0605ef3f84 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
+ WRITE_STRING_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index dd72a9fc3c..52d92df25d 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
+%type <str> optColumnCompression
+
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
* They must be listed first so that their numeric codes do not depend on
@@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P
CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE
CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT
- COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT
- CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE
- CROSS CSV CUBE CURRENT_P
+ COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT
+ CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY
+ COST CREATE CROSS CSV CUBE CURRENT_P
CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA
CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE
@@ -3421,11 +3423,12 @@ TypedTableElement:
| TableConstraint { $$ = $1; }
;
-columnDef: ColId Typename create_generic_options ColQualList
+columnDef: ColId Typename optColumnCompression create_generic_options ColQualList
{
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
+ n->compression = $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList
n->raw_default = NULL;
n->cooked_default = NULL;
n->collOid = InvalidOid;
- n->fdwoptions = $3;
- SplitColQualList($4, &n->constraints, &n->collClause,
+ n->fdwoptions = $4;
+ SplitColQualList($5, &n->constraints, &n->collClause,
yyscanner);
n->location = @1;
$$ = (Node *)n;
@@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList
}
;
+optColumnCompression:
+ COMPRESSION name
+ {
+ $$ = $2;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
| /*EMPTY*/ { $$ = NIL; }
@@ -3710,6 +3721,7 @@ TableLikeOption:
| INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; }
| STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; }
| STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; }
+ | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; }
| ALL { $$ = CREATE_TABLE_LIKE_ALL; }
;
@@ -15285,6 +15297,7 @@ unreserved_keyword:
| COMMENTS
| COMMIT
| COMMITTED
+ | COMPRESSION
| CONFIGURATION
| CONFLICT
| CONNECTION
@@ -15805,6 +15818,7 @@ bare_label_keyword:
| COMMENTS
| COMMIT
| COMMITTED
+ | COMPRESSION
| CONCURRENTLY
| CONFIGURATION
| CONFLICT
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index b31f3afa03..cf4413da64 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -27,6 +27,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "access/relation.h"
#include "access/reloptions.h"
@@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
else
def->storage = 0;
+ /* Likewise, copy compression if requested */
+ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
+ && OidIsValid(attribute->attcompression))
+ def->compression = get_am_name(attribute->attcompression);
+ else
+ def->compression = NULL;
+
/* Likewise, copy comment if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) &&
(comment = GetComment(attribute->attrelid,
diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c
index c2f910d606..fe133c76c2 100644
--- a/src/backend/utils/adt/pseudotypes.c
+++ b/src/backend/utils/adt/pseudotypes.c
@@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler);
PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler);
PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler);
PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler);
+PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler);
PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler);
PSEUDOTYPE_DUMMY_IO_FUNCS(internal);
PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement);
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 479ed9ae54..a35abe5f1a 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -17,9 +17,11 @@
#include <ctype.h>
#include <limits.h>
+#include "access/compressamapi.h"
#include "access/detoast.h"
#include "catalog/pg_collation.h"
#include "catalog/pg_type.h"
+#include "commands/defrem.h"
#include "common/hashfn.h"
#include "common/hex.h"
#include "common/int.h"
@@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS)
PG_RETURN_INT32(result);
}
+/*
+ * Return the compression method stored in the compressed attribute. Return
+ * NULL for non varlena type or the uncompressed data.
+ */
+Datum
+pg_column_compression(PG_FUNCTION_ARGS)
+{
+ Datum value = PG_GETARG_DATUM(0);
+ int typlen;
+ Oid cmoid;
+
+ /* On first call, get the input type's typlen, and save at *fn_extra */
+ if (fcinfo->flinfo->fn_extra == NULL)
+ {
+ /* Lookup the datatype of the supplied argument */
+ Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0);
+
+ typlen = get_typlen(argtypeid);
+ if (typlen == 0) /* should not happen */
+ elog(ERROR, "cache lookup failed for type %u", argtypeid);
+
+ fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt,
+ sizeof(int));
+ *((int *) fcinfo->flinfo->fn_extra) = typlen;
+ }
+ else
+ typlen = *((int *) fcinfo->flinfo->fn_extra);
+
+ if (typlen != -1)
+ PG_RETURN_NULL();
+
+ cmoid =
+ toast_get_compression_oid((struct varlena *) DatumGetPointer(value));
+
+ if (!OidIsValid(cmoid))
+ PG_RETURN_NULL();
+ else
+ PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid)));
+}
+
/*
* string_agg - Concatenates values and returns string.
*
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index eea9f30a79..7332f93a25 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -160,6 +160,7 @@ typedef struct _dumpOptions
int no_subscriptions;
int no_synchronized_snapshots;
int no_unlogged_table_data;
+ int no_compression_methods;
int serializable_deferrable;
int disable_triggers;
int outputNoTablespaces;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index eb988d7eb4..46044cb92a 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -387,6 +387,7 @@ main(int argc, char **argv)
{"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1},
{"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1},
{"no-subscriptions", no_argument, &dopt.no_subscriptions, 1},
+ {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1},
{"no-sync", no_argument, NULL, 7},
{"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1},
{"rows-per-insert", required_argument, NULL, 10},
@@ -1047,6 +1048,7 @@ help(const char *progname)
printf(_(" --no-publications do not dump publications\n"));
printf(_(" --no-security-labels do not dump security label assignments\n"));
printf(_(" --no-subscriptions do not dump subscriptions\n"));
+ printf(_(" --no-compression-methods do not dump compression methods\n"));
printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n"));
printf(_(" --no-tablespaces do not dump tablespace assignments\n"));
printf(_(" --no-unlogged-table-data do not dump unlogged table data\n"));
@@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
{
DumpOptions *dopt = fout->dopt;
PQExpBuffer q = createPQExpBuffer();
+ bool createWithCompression;
for (int i = 0; i < numTables; i++)
{
@@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
appendPQExpBufferStr(q,
"'' AS attidentity,\n");
+ createWithCompression = (fout->remoteVersion >= 140000);
+
+ if (createWithCompression)
+ appendPQExpBuffer(q,
+ "am.amname AS attcmname,\n");
+ else
+ appendPQExpBuffer(q,
+ "NULL AS attcmname,\n");
+
if (fout->remoteVersion >= 110000)
appendPQExpBufferStr(q,
"CASE WHEN a.atthasmissing AND NOT a.attisdropped "
@@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
/* need left join here to not fail on dropped columns ... */
appendPQExpBuffer(q,
"FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t "
- "ON a.atttypid = t.oid\n"
+ "ON a.atttypid = t.oid\n");
+
+ if (createWithCompression)
+ appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am "
+ "ON a.attcompression = am.oid\n");
+ appendPQExpBuffer(q,
"WHERE a.attrelid = '%u'::pg_catalog.oid "
"AND a.attnum > 0::pg_catalog.int2\n"
"ORDER BY a.attnum",
@@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid));
tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *));
tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *));
+ tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *));
tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *));
@@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation")));
tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions")));
tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval")));
+ tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname")));
tbinfo->attrdefs[j] = NULL; /* fix below */
if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't')
hasdefaults = true;
@@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
{
bool print_default;
bool print_notnull;
+ bool has_non_default_compression;
/*
* Default value --- suppress if to be printed separately.
@@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
!dopt->binary_upgrade)
continue;
+ has_non_default_compression = (tbinfo->attcmnames[j] &&
+ (strcmp(tbinfo->attcmnames[j], "pglz") != 0));
+
/* Format properly if not first attr */
if (actual_atts == 0)
appendPQExpBufferStr(q, " (");
@@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
tbinfo->atttypnames[j]);
}
+ /*
+ * Attribute compression
+ */
+ if (!dopt->no_compression_methods &&
+ tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) &&
+ (has_non_default_compression || dopt->binary_upgrade))
+ {
+ appendPQExpBuffer(q, " COMPRESSION %s",
+ tbinfo->attcmnames[j]);
+ }
+
if (print_default)
{
if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 0a2213fb06..1789e18f46 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -326,6 +326,7 @@ typedef struct _tableInfo
char *partbound; /* partition bound definition */
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
+ char **attcmnames; /* per-attribute current compression method */
/*
* Stuff computed only for dumpable tables.
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 737e46464a..97791f8dbe 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -2284,9 +2284,9 @@ my %tests = (
regexp => qr/^
\QCREATE TABLE dump_test.test_table (\E\n
\s+\Qcol1 integer NOT NULL,\E\n
- \s+\Qcol2 text,\E\n
- \s+\Qcol3 text,\E\n
- \s+\Qcol4 text,\E\n
+ \s+\Qcol2 text\E\D*,\n
+ \s+\Qcol3 text\E\D*,\n
+ \s+\Qcol4 text\E\D*,\n
\s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n
\Q)\E\n
\QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm,
@@ -2326,7 +2326,7 @@ my %tests = (
regexp => qr/^
\QCREATE TABLE dump_test.test_second_table (\E
\n\s+\Qcol1 integer,\E
- \n\s+\Qcol2 text\E
+ \n\s+\Qcol2 text\E\D*
\n\);
/xm,
like =>
@@ -2441,7 +2441,7 @@ my %tests = (
\n\s+\Qcol1 integer,\E
\n\s+\Qcol2 boolean,\E
\n\s+\Qcol3 boolean,\E
- \n\s+\Qcol4 bit(5),\E
+ \n\s+\Qcol4 bit(5)\E\D*,
\n\s+\Qcol5 double precision\E
\n\);
/xm,
@@ -2459,7 +2459,7 @@ my %tests = (
regexp => qr/^
\QCREATE TABLE dump_test.test_table_identity (\E\n
\s+\Qcol1 integer NOT NULL,\E\n
- \s+\Qcol2 text\E\n
+ \s+\Qcol2 text\E\D*\n
\);
.*
\QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 20af5a92b4..ba464d463e 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose)
" CASE amtype"
" WHEN 'i' THEN '%s'"
" WHEN 't' THEN '%s'"
+ " WHEN 'c' THEN '%s'"
" END AS \"%s\"",
gettext_noop("Name"),
gettext_noop("Index"),
gettext_noop("Table"),
+ gettext_noop("Compression"),
gettext_noop("Type"));
if (verbose)
@@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname,
bool printTableInitialized = false;
int i;
char *view_def = NULL;
- char *headers[11];
+ char *headers[12];
PQExpBufferData title;
PQExpBufferData tmpbuf;
int cols;
@@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname,
fdwopts_col = -1,
attstorage_col = -1,
attstattarget_col = -1,
- attdescr_col = -1;
+ attdescr_col = -1,
+ attcompression_col = -1;
int numrows;
struct
{
@@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname,
appendPQExpBufferStr(&buf, ",\n a.attstorage");
attstorage_col = cols++;
+ /* compresssion info */
+ if (pset.sversion >= 140000 &&
+ (tableinfo.relkind == RELKIND_RELATION ||
+ tableinfo.relkind == RELKIND_PARTITIONED_TABLE ||
+ tableinfo.relkind == RELKIND_MATVIEW))
+ {
+ appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE "
+ " (SELECT am.amname "
+ " FROM pg_catalog.pg_am am "
+ " WHERE am.oid = a.attcompression) "
+ " END AS attcompression");
+ attcompression_col = cols++;
+ }
+
/* stats target, if relevant to relkind */
if (tableinfo.relkind == RELKIND_RELATION ||
tableinfo.relkind == RELKIND_INDEX ||
@@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname,
headers[cols++] = gettext_noop("FDW options");
if (attstorage_col >= 0)
headers[cols++] = gettext_noop("Storage");
+ if (attcompression_col >= 0)
+ headers[cols++] = gettext_noop("Compression");
if (attstattarget_col >= 0)
headers[cols++] = gettext_noop("Stats target");
if (attdescr_col >= 0)
@@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname,
false, false);
}
+ /* Column compression */
+ if (attcompression_col >= 0)
+ printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col),
+ false, false);
+
/* Statistics target, if the relkind supports this feature */
if (attstattarget_col >= 0)
printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col),
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
new file mode 100644
index 0000000000..5a8e23d926
--- /dev/null
+++ b/src/include/access/compressamapi.h
@@ -0,0 +1,99 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressamapi.h
+ * API for Postgres compression methods.
+ *
+ * Portions Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ * src/include/access/compressamapi.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef COMPRESSAMAPI_H
+#define COMPRESSAMAPI_H
+
+#include "postgres.h"
+
+#include "catalog/pg_am_d.h"
+#include "nodes/nodes.h"
+
+/*
+ * Built-in compression method-id. The toast compression header will store
+ * this in the first 2 bits of the raw length. These built-in compression
+ * method-id are directly mapped to the built-in compression method oid.
+ */
+typedef enum CompressionId
+{
+ PGLZ_COMPRESSION_ID = 0,
+ LZ4_COMPRESSION_ID = 1
+} CompressionId;
+
+/* Use default compression method if it is not specified. */
+#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID
+#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
+ (storage) != TYPSTORAGE_EXTERNAL)
+/* compression handler routines */
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmdecompress_slice_function)
+ (const struct varlena *value, int32 slicelength);
+
+/*
+ * API struct for a compression AM.
+ *
+ * 'datum_compress' - varlena compression function.
+ * 'datum_decompress' - varlena decompression function.
+ * 'datum_decompress_slice' - varlena slice decompression functions.
+ */
+typedef struct CompressionAmRoutine
+{
+ NodeTag type;
+
+ cmcompress_function datum_compress;
+ cmdecompress_function datum_decompress;
+ cmdecompress_slice_function datum_decompress_slice;
+} CompressionAmRoutine;
+
+extern const CompressionAmRoutine pglz_compress_methods;
+extern const CompressionAmRoutine lz4_compress_methods;
+
+/*
+ * CompressionOidToId - Convert compression Oid to built-in compression id.
+ *
+ * For more details refer comment atop CompressionId in compressamapi.h
+ */
+static inline CompressionId
+CompressionOidToId(Oid cmoid)
+{
+ switch (cmoid)
+ {
+ case PGLZ_COMPRESSION_AM_OID:
+ return PGLZ_COMPRESSION_ID;
+ case LZ4_COMPRESSION_AM_OID:
+ return LZ4_COMPRESSION_ID;
+ default:
+ elog(ERROR, "invalid compression method oid %u", cmoid);
+ }
+}
+
+/*
+ * CompressionIdToOid - Convert built-in compression id to Oid
+ *
+ * For more details refer comment atop CompressionId in compressamapi.h
+ */
+static inline Oid
+CompressionIdToOid(CompressionId cmid)
+{
+ switch (cmid)
+ {
+ case PGLZ_COMPRESSION_ID:
+ return PGLZ_COMPRESSION_AM_OID;
+ case LZ4_COMPRESSION_ID:
+ return LZ4_COMPRESSION_AM_OID;
+ default:
+ elog(ERROR, "invalid compression method id %d", cmid);
+ }
+}
+
+#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h
index 0adf53c77b..6cdc37542d 100644
--- a/src/include/access/detoast.h
+++ b/src/include/access/detoast.h
@@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value);
*/
extern Size toast_datum_size(Datum value);
+/* ----------
+ * toast_get_compression_oid -
+ *
+ * Return the compression method oid from the compressed value
+ * ----------
+ */
+extern Oid toast_get_compression_oid(struct varlena *attr);
+
#endif /* DETOAST_H */
diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h
index a9a6d644bc..dca0bc37f3 100644
--- a/src/include/access/toast_helper.h
+++ b/src/include/access/toast_helper.h
@@ -32,6 +32,7 @@ typedef struct
struct varlena *tai_oldexternal;
int32 tai_size;
uint8 tai_colflags;
+ Oid tai_compression;
} ToastAttrInfo;
/*
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index cedfb890d8..31ff91a09c 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -12,6 +12,7 @@
#ifndef TOAST_INTERNALS_H
#define TOAST_INTERNALS_H
+#include "access/compressamapi.h"
#include "storage/lockdefs.h"
#include "utils/relcache.h"
#include "utils/snapshot.h"
@@ -22,22 +23,22 @@
typedef struct toast_compress_header
{
int32 vl_len_; /* varlena header (do not touch directly!) */
- int32 rawsize;
+ uint32 info; /* 2 bits for compression method and 30 bits
+ * rawsize */
} toast_compress_header;
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
-#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
-#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize)
-#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ)
-#define TOAST_COMPRESS_RAWDATA(ptr) \
- (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ)
-#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \
- (((toast_compress_header *) (ptr))->rawsize = (len))
+#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
+#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
+ do { \
+ Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \
+ ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
+ } while (0)
-extern Datum toast_compress_datum(Datum value);
+extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative);
diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat
index 6082f0e6a8..605684408c 100644
--- a/src/include/catalog/pg_am.dat
+++ b/src/include/catalog/pg_am.dat
@@ -33,5 +33,11 @@
{ oid => '3580', oid_symbol => 'BRIN_AM_OID',
descr => 'block range index (BRIN) access method',
amname => 'brin', amhandler => 'brinhandler', amtype => 'i' },
+{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID',
+ descr => 'pglz compression access method',
+ amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' },
+{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID',
+ descr => 'lz4 compression access method',
+ amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' },
]
diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h
index ced86faef8..65079f3821 100644
--- a/src/include/catalog/pg_am.h
+++ b/src/include/catalog/pg_am.h
@@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op
*/
#define AMTYPE_INDEX 'i' /* index access method */
#define AMTYPE_TABLE 't' /* table access method */
+#define AMTYPE_COMPRESSION 'c' /* compression access method */
#endif /* EXPOSE_TO_CLIENT_CODE */
diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h
index 3db42abf08..797e78b17c 100644
--- a/src/include/catalog/pg_attribute.h
+++ b/src/include/catalog/pg_attribute.h
@@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75,
/* attribute's collation, if any */
Oid attcollation BKI_LOOKUP_OPT(pg_collation);
+ /*
+ * OID of compression AM. Must be InvalidOid if and only if typstorage is
+ * 'plain' or 'external'.
+ */
+ Oid attcompression BKI_DEFAULT(0);
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* NOTE: The following fields are not present in tuple descriptors. */
@@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75,
* can access fields beyond attcollation except in a real tuple!
*/
#define ATTRIBUTE_FIXED_PART_SIZE \
- (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid))
+ (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid))
/* ----------------
* Form_pg_attribute corresponds to a pointer to a tuple with
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4e0c9be58c..4a13844ce6 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -941,6 +941,15 @@
prorettype => 'void', proargtypes => 'regclass int8',
prosrc => 'brin_desummarize_range' },
+{ oid => '6015', descr => 'pglz compression access method handler',
+ proname => 'pglzhandler', provolatile => 'v',
+ prorettype => 'compression_am_handler', proargtypes => 'internal',
+ prosrc => 'pglzhandler' },
+{ oid => '6016', descr => 'lz4 compression access method handler',
+ proname => 'lz4handler', provolatile => 'v',
+ prorettype => 'compression_am_handler', proargtypes => 'internal',
+ prosrc => 'lz4handler' },
+
{ oid => '338', descr => 'validate an operator class',
proname => 'amvalidate', provolatile => 'v', prorettype => 'bool',
proargtypes => 'oid', prosrc => 'amvalidate' },
@@ -7095,6 +7104,10 @@
descr => 'bytes required to store the value, perhaps with compression',
proname => 'pg_column_size', provolatile => 's', prorettype => 'int4',
proargtypes => 'any', prosrc => 'pg_column_size' },
+{ oid => '2121',
+ descr => 'compression method for the compressed datum',
+ proname => 'pg_column_compression', provolatile => 's', prorettype => 'text',
+ proargtypes => 'any', prosrc => 'pg_column_compression' },
{ oid => '2322',
descr => 'total disk space usage for the specified tablespace',
proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8',
@@ -7265,6 +7278,13 @@
{ oid => '268', descr => 'I/O',
proname => 'table_am_handler_out', prorettype => 'cstring',
proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' },
+{ oid => '560', descr => 'I/O',
+ proname => 'compression_am_handler_in', proisstrict => 'f',
+ prorettype => 'compression_am_handler', proargtypes => 'cstring',
+ prosrc => 'compression_am_handler_in' },
+{ oid => '561', descr => 'I/O',
+ proname => 'compression_am_handler_out', prorettype => 'cstring',
+ proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' },
{ oid => '5086', descr => 'I/O',
proname => 'anycompatible_in', prorettype => 'anycompatible',
proargtypes => 'cstring', prosrc => 'anycompatible_in' },
diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat
index 8959c2f53b..306aabf6c1 100644
--- a/src/include/catalog/pg_type.dat
+++ b/src/include/catalog/pg_type.dat
@@ -626,6 +626,11 @@
typcategory => 'P', typinput => 'index_am_handler_in',
typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-',
typalign => 'i' },
+ { oid => '5559',
+ typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p',
+ typcategory => 'P', typinput => 'compression_am_handler_in',
+ typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-',
+ typalign => 'i' },
{ oid => '3310',
descr => 'pseudo-type for the result of a tablesample method function',
typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p',
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index 1a79540c94..e5aea8a240 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId,
extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt);
extern Oid get_index_am_oid(const char *amname, bool missing_ok);
extern Oid get_table_am_oid(const char *amname, bool missing_ok);
+extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index 071e363d54..6495162a33 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd);
extern void CheckSubscriptionRelkind(char relkind, const char *nspname,
const char *relname);
-
+extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
+ TupleTableSlot **outslot,
+ TupleDesc targetTupDesc);
#endif /* EXECUTOR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index b6a88ff76b..18e8ecfd90 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1182,6 +1182,12 @@ typedef struct ModifyTableState
*/
TupleTableSlot *mt_root_tuple_slot;
+ /*
+ * Slot for storing the modified tuple, incase the target attribute's
+ * compression method doesn't match with the source table.
+ */
+ TupleTableSlot *mt_decompress_tuple_slot;
+
/* Tuple-routing support info */
struct PartitionTupleRouting *mt_partition_tuple_routing;
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 40ae489c23..20d6f96f62 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -512,6 +512,7 @@ typedef enum NodeTag
T_IndexAmRoutine, /* in access/amapi.h */
T_TableAmRoutine, /* in access/tableam.h */
T_TsmRoutine, /* in access/tsmapi.h */
+ T_CompressionAmRoutine, /* in access/compressamapi.h */
T_ForeignKeyCacheInfo, /* in utils/rel.h */
T_CallContext, /* in nodes/parsenodes.h */
T_SupportRequestSimplify, /* in nodes/supportnodes.h */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 236832a2ca..19d2ba26bf 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -646,6 +646,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
+ char *compression; /* compression method for column */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
@@ -685,6 +686,7 @@ typedef enum TableLikeOption
CREATE_TABLE_LIKE_INDEXES = 1 << 5,
CREATE_TABLE_LIKE_STATISTICS = 1 << 6,
CREATE_TABLE_LIKE_STORAGE = 1 << 7,
+ CREATE_TABLE_LIKE_COMPRESSION = 1 << 8,
CREATE_TABLE_LIKE_ALL = PG_INT32_MAX
} TableLikeOption;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..ca1f950cbe 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 55cab4d2bf..53d378b67d 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -346,6 +346,9 @@
/* Define to 1 if you have the `z' library (-lz). */
#undef HAVE_LIBZ
+/* Define to 1 if you have the `lz4' library (-llz4). */
+#undef HAVE_LIBLZ4
+
/* Define to 1 if you have the `link' function. */
#undef HAVE_LINK
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 2ed572004d..667927fd7c 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -145,7 +145,8 @@ typedef union
struct /* Compressed-in-line format */
{
uint32 va_header;
- uint32 va_rawsize; /* Original data size (excludes header) */
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
} varattrib_4b;
@@ -274,14 +275,21 @@ typedef struct
(VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT)
#define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data)
+#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data)
#define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data)
#define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data)
#define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data)
#define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data)
+#define VARLENA_RAWSIZE_BITS 30
+#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1)
+
+/* va_info in va_compress contains raw size of datum and optional flags */
#define VARRAWSIZE_4B_C(PTR) \
- (((varattrib_4b *) (PTR))->va_compressed.va_rawsize)
+ (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK)
+#define VARFLAGS_4B_C(PTR) \
+ (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS)
/* Externally visible macros */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
new file mode 100644
index 0000000000..167878e78b
--- /dev/null
+++ b/src/test/regress/expected/compression.out
@@ -0,0 +1,201 @@
+\set HIDE_COMPRESSAM off
+-- test creating table with compression method
+CREATE TABLE cmdata(f1 text COMPRESSION pglz);
+CREATE INDEX idx ON cmdata(f1);
+INSERT INTO cmdata VALUES(repeat('1234567890',1000));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+------+-----------+----------+---------+----------+--------------+-------------
+ f1 | text | | | | extended | |
+Indexes:
+ "idx" btree (f1)
+
+CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4);
+INSERT INTO cmdata1 VALUES(repeat('1234567890',1004));
+\d+ cmdata1
+ Table "public.cmdata1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+------+-----------+----------+---------+----------+--------------+-------------
+ f1 | text | | | | extended | |
+
+-- try setting compression for incompressible data type
+CREATE TABLE cmdata2 (f1 int COMPRESSION pglz);
+ERROR: column data type integer does not support compression
+-- verify stored compression method
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+SELECT pg_column_compression(f1) FROM cmdata1;
+ pg_column_compression
+-----------------------
+ lz4
+(1 row)
+
+-- decompress data slice
+SELECT SUBSTR(f1, 200, 5) FROM cmdata;
+ substr
+--------
+ 01234
+(1 row)
+
+SELECT SUBSTR(f1, 2000, 50) FROM cmdata1;
+ substr
+----------------------------------------------------
+ 01234567890123456789012345678901234567890123456789
+(1 row)
+
+-- copy with table creation
+SELECT * INTO cmmove1 FROM cmdata;
+SELECT pg_column_compression(f1) FROM cmmove1;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+-- update using datum from different table
+CREATE TABLE cmmove2(f1 text COMPRESSION pglz);
+INSERT INTO cmmove2 VALUES (repeat('1234567890',1004));
+SELECT pg_column_compression(f1) FROM cmmove2;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata;
+SELECT pg_column_compression(f1) FROM cmmove2;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1;
+SELECT pg_column_compression(f1) FROM cmmove2;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+-- copy to existing table
+CREATE TABLE cmmove3(f1 text COMPRESSION pglz);
+INSERT INTO cmmove3 SELECT * FROM cmdata;
+INSERT INTO cmmove3 SELECT * FROM cmdata1;
+SELECT pg_column_compression(f1) FROM cmmove2;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+-- test external compressed data
+CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS
+'select array_agg(md5(g::text))::text from generate_series(1, 256) g';
+CREATE TABLE cmdata2 (f1 text COMPRESSION pglz);
+INSERT INTO cmdata2 select large_val() || repeat('a', 4000);
+SELECT pg_column_compression(f1) FROM cmdata2;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+INSERT INTO cmdata1 SELECT * FROM cmdata2;
+SELECT pg_column_compression(f1) FROM cmdata1;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+(2 rows)
+
+DROP TABLE cmdata2;
+-- test LIKE INCLUDING COMPRESSION
+CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION);
+\d+ cmdata2
+ Table "public.cmdata2"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+------+-----------+----------+---------+----------+--------------+-------------
+ f1 | text | | | | extended | |
+
+-- test compression with materialized view
+CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1;
+\d+ mv
+ Materialized view "public.mv"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+------+-----------+----------+---------+----------+--------------+-------------
+ x | text | | | | extended | |
+View definition:
+ SELECT cmdata1.f1 AS x
+ FROM cmdata1;
+
+SELECT pg_column_compression(f1) FROM cmdata1;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+(2 rows)
+
+SELECT pg_column_compression(x) FROM mv;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+-- test compression with partition
+CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1);
+CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0);
+CREATE TABLE cmpart2(f1 text COMPRESSION pglz);
+ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1);
+INSERT INTO cmpart VALUES (repeat('123456789',1004));
+INSERT INTO cmpart VALUES (repeat('123456789',4004));
+SELECT pg_column_compression(f1) FROM cmpart;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+-- test compression with inheritence, error
+CREATE TABLE cminh() INHERITS(cmdata, cmdata1);
+NOTICE: merging multiple inherited definitions of column "f1"
+ERROR: column "f1" has a compression method conflict
+DETAIL: pglz versus lz4
+CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata);
+NOTICE: merging column "f1" with inherited definition
+ERROR: column "f1" has a compression method conflict
+DETAIL: pglz versus lz4
+-- check data is ok
+SELECT length(f1) FROM cmdata;
+ length
+--------
+ 10000
+(1 row)
+
+SELECT length(f1) FROM cmdata1;
+ length
+--------
+ 10040
+ 12449
+(2 rows)
+
+SELECT length(f1) FROM cmmove1;
+ length
+--------
+ 10000
+(1 row)
+
+SELECT length(f1) FROM cmmove2;
+ length
+--------
+ 10040
+(1 row)
+
+SELECT length(f1) FROM cmmove3;
+ length
+--------
+ 10000
+ 10040
+(2 rows)
+
+\set HIDE_COMPRESSAM on
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
new file mode 100644
index 0000000000..329e7881b0
--- /dev/null
+++ b/src/test/regress/expected/compression_1.out
@@ -0,0 +1,189 @@
+\set HIDE_COMPRESSAM off
+-- test creating table with compression method
+CREATE TABLE cmdata(f1 text COMPRESSION pglz);
+CREATE INDEX idx ON cmdata(f1);
+INSERT INTO cmdata VALUES(repeat('1234567890',1000));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+------+-----------+----------+---------+----------+--------------+-------------
+ f1 | text | | | | extended | |
+Indexes:
+ "idx" btree (f1)
+
+CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4);
+ERROR: not built with lz4 support
+INSERT INTO cmdata1 VALUES(repeat('1234567890',1004));
+ERROR: relation "cmdata1" does not exist
+LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004));
+ ^
+\d+ cmdata1
+-- try setting compression for incompressible data type
+CREATE TABLE cmdata2 (f1 int COMPRESSION pglz);
+ERROR: column data type integer does not support compression
+-- verify stored compression method
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+SELECT pg_column_compression(f1) FROM cmdata1;
+ERROR: relation "cmdata1" does not exist
+LINE 1: SELECT pg_column_compression(f1) FROM cmdata1;
+ ^
+-- decompress data slice
+SELECT SUBSTR(f1, 200, 5) FROM cmdata;
+ substr
+--------
+ 01234
+(1 row)
+
+SELECT SUBSTR(f1, 2000, 50) FROM cmdata1;
+ERROR: relation "cmdata1" does not exist
+LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1;
+ ^
+-- copy with table creation
+SELECT * INTO cmmove1 FROM cmdata;
+SELECT pg_column_compression(f1) FROM cmmove1;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+-- update using datum from different table
+CREATE TABLE cmmove2(f1 text COMPRESSION pglz);
+INSERT INTO cmmove2 VALUES (repeat('1234567890',1004));
+SELECT pg_column_compression(f1) FROM cmmove2;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata;
+SELECT pg_column_compression(f1) FROM cmmove2;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1;
+ERROR: relation "cmdata1" does not exist
+LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1;
+ ^
+SELECT pg_column_compression(f1) FROM cmmove2;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+-- copy to existing table
+CREATE TABLE cmmove3(f1 text COMPRESSION pglz);
+INSERT INTO cmmove3 SELECT * FROM cmdata;
+INSERT INTO cmmove3 SELECT * FROM cmdata1;
+ERROR: relation "cmdata1" does not exist
+LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1;
+ ^
+SELECT pg_column_compression(f1) FROM cmmove2;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+-- test external compressed data
+CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS
+'select array_agg(md5(g::text))::text from generate_series(1, 256) g';
+CREATE TABLE cmdata2 (f1 text COMPRESSION pglz);
+INSERT INTO cmdata2 select large_val() || repeat('a', 4000);
+SELECT pg_column_compression(f1) FROM cmdata2;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+INSERT INTO cmdata1 SELECT * FROM cmdata2;
+ERROR: relation "cmdata1" does not exist
+LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2;
+ ^
+SELECT pg_column_compression(f1) FROM cmdata1;
+ERROR: relation "cmdata1" does not exist
+LINE 1: SELECT pg_column_compression(f1) FROM cmdata1;
+ ^
+DROP TABLE cmdata2;
+-- test LIKE INCLUDING COMPRESSION
+CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION);
+ERROR: relation "cmdata1" does not exist
+LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION);
+ ^
+\d+ cmdata2
+-- test compression with materialized view
+CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1;
+ERROR: relation "cmdata1" does not exist
+LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1;
+ ^
+\d+ mv
+SELECT pg_column_compression(f1) FROM cmdata1;
+ERROR: relation "cmdata1" does not exist
+LINE 1: SELECT pg_column_compression(f1) FROM cmdata1;
+ ^
+SELECT pg_column_compression(x) FROM mv;
+ERROR: relation "mv" does not exist
+LINE 1: SELECT pg_column_compression(x) FROM mv;
+ ^
+-- test compression with partition
+CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1);
+ERROR: not built with lz4 support
+CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0);
+ERROR: relation "cmpart" does not exist
+CREATE TABLE cmpart2(f1 text COMPRESSION pglz);
+ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1);
+ERROR: relation "cmpart" does not exist
+INSERT INTO cmpart VALUES (repeat('123456789',1004));
+ERROR: relation "cmpart" does not exist
+LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004));
+ ^
+INSERT INTO cmpart VALUES (repeat('123456789',4004));
+ERROR: relation "cmpart" does not exist
+LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004));
+ ^
+SELECT pg_column_compression(f1) FROM cmpart;
+ERROR: relation "cmpart" does not exist
+LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
+ ^
+-- test compression with inheritence, error
+CREATE TABLE cminh() INHERITS(cmdata, cmdata1);
+ERROR: relation "cmdata1" does not exist
+CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata);
+NOTICE: merging column "f1" with inherited definition
+ERROR: column "f1" has a compression method conflict
+DETAIL: pglz versus lz4
+-- check data is ok
+SELECT length(f1) FROM cmdata;
+ length
+--------
+ 10000
+(1 row)
+
+SELECT length(f1) FROM cmdata1;
+ERROR: relation "cmdata1" does not exist
+LINE 1: SELECT length(f1) FROM cmdata1;
+ ^
+SELECT length(f1) FROM cmmove1;
+ length
+--------
+ 10000
+(1 row)
+
+SELECT length(f1) FROM cmmove2;
+ length
+--------
+ 10000
+(1 row)
+
+SELECT length(f1) FROM cmmove3;
+ length
+--------
+ 10000
+(1 row)
+
+\set HIDE_COMPRESSAM on
diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out
index 7204fdb0b4..14f3fa3fc8 100644
--- a/src/test/regress/expected/psql.out
+++ b/src/test/regress/expected/psql.out
@@ -4898,8 +4898,8 @@ Indexes:
-- check printing info about access methods
\dA
List of access methods
- Name | Type
---------+-------
+ Name | Type
+--------+-------------
brin | Index
btree | Index
gin | Index
@@ -4907,13 +4907,15 @@ List of access methods
hash | Index
heap | Table
heap2 | Table
+ lz4 | Compression
+ pglz | Compression
spgist | Index
-(8 rows)
+(10 rows)
\dA *
List of access methods
- Name | Type
---------+-------
+ Name | Type
+--------+-------------
brin | Index
btree | Index
gin | Index
@@ -4921,8 +4923,10 @@ List of access methods
hash | Index
heap | Table
heap2 | Table
+ lz4 | Compression
+ pglz | Compression
spgist | Index
-(8 rows)
+(10 rows)
\dA h*
List of access methods
@@ -4947,32 +4951,36 @@ List of access methods
\dA: extra argument "bar" ignored
\dA+
- List of access methods
- Name | Type | Handler | Description
---------+-------+----------------------+----------------------------------------
- brin | Index | brinhandler | block range index (BRIN) access method
- btree | Index | bthandler | b-tree index access method
- gin | Index | ginhandler | GIN index access method
- gist | Index | gisthandler | GiST index access method
- hash | Index | hashhandler | hash index access method
- heap | Table | heap_tableam_handler | heap table access method
- heap2 | Table | heap_tableam_handler |
- spgist | Index | spghandler | SP-GiST index access method
-(8 rows)
+ List of access methods
+ Name | Type | Handler | Description
+--------+-------------+----------------------+----------------------------------------
+ brin | Index | brinhandler | block range index (BRIN) access method
+ btree | Index | bthandler | b-tree index access method
+ gin | Index | ginhandler | GIN index access method
+ gist | Index | gisthandler | GiST index access method
+ hash | Index | hashhandler | hash index access method
+ heap | Table | heap_tableam_handler | heap table access method
+ heap2 | Table | heap_tableam_handler |
+ lz4 | Compression | lz4handler | lz4 compression access method
+ pglz | Compression | pglzhandler | pglz compression access method
+ spgist | Index | spghandler | SP-GiST index access method
+(10 rows)
\dA+ *
- List of access methods
- Name | Type | Handler | Description
---------+-------+----------------------+----------------------------------------
- brin | Index | brinhandler | block range index (BRIN) access method
- btree | Index | bthandler | b-tree index access method
- gin | Index | ginhandler | GIN index access method
- gist | Index | gisthandler | GiST index access method
- hash | Index | hashhandler | hash index access method
- heap | Table | heap_tableam_handler | heap table access method
- heap2 | Table | heap_tableam_handler |
- spgist | Index | spghandler | SP-GiST index access method
-(8 rows)
+ List of access methods
+ Name | Type | Handler | Description
+--------+-------------+----------------------+----------------------------------------
+ brin | Index | brinhandler | block range index (BRIN) access method
+ btree | Index | bthandler | b-tree index access method
+ gin | Index | ginhandler | GIN index access method
+ gist | Index | gisthandler | GiST index access method
+ hash | Index | hashhandler | hash index access method
+ heap | Table | heap_tableam_handler | heap table access method
+ heap2 | Table | heap_tableam_handler |
+ lz4 | Compression | lz4handler | lz4 compression access method
+ pglz | Compression | pglzhandler | pglz compression access method
+ spgist | Index | spghandler | SP-GiST index access method
+(10 rows)
\dA+ h*
List of access methods
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 12bb67e491..e9c0fc9760 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr
# ----------
# Another group of parallel tests
# ----------
-test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain
+test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression
# event triggers cannot run concurrently with any test that runs DDL
# oidjoins is read-only, though, and should run late for best coverage
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index 59b416fd80..4d5577bae3 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -199,6 +199,7 @@ test: partition_aggregate
test: partition_info
test: tuplesort
test: explain
+test: compression
test: event_trigger
test: oidjoins
test: fast_default
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
new file mode 100644
index 0000000000..450416ecb4
--- /dev/null
+++ b/src/test/regress/sql/compression.sql
@@ -0,0 +1,85 @@
+\set HIDE_COMPRESSAM off
+
+-- test creating table with compression method
+CREATE TABLE cmdata(f1 text COMPRESSION pglz);
+CREATE INDEX idx ON cmdata(f1);
+INSERT INTO cmdata VALUES(repeat('1234567890',1000));
+\d+ cmdata
+
+CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4);
+INSERT INTO cmdata1 VALUES(repeat('1234567890',1004));
+\d+ cmdata1
+
+-- try setting compression for incompressible data type
+CREATE TABLE cmdata2 (f1 int COMPRESSION pglz);
+
+-- verify stored compression method
+SELECT pg_column_compression(f1) FROM cmdata;
+SELECT pg_column_compression(f1) FROM cmdata1;
+
+-- decompress data slice
+SELECT SUBSTR(f1, 200, 5) FROM cmdata;
+SELECT SUBSTR(f1, 2000, 50) FROM cmdata1;
+
+-- copy with table creation
+SELECT * INTO cmmove1 FROM cmdata;
+SELECT pg_column_compression(f1) FROM cmmove1;
+
+-- update using datum from different table
+CREATE TABLE cmmove2(f1 text COMPRESSION pglz);
+INSERT INTO cmmove2 VALUES (repeat('1234567890',1004));
+SELECT pg_column_compression(f1) FROM cmmove2;
+
+UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata;
+SELECT pg_column_compression(f1) FROM cmmove2;
+UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1;
+SELECT pg_column_compression(f1) FROM cmmove2;
+
+-- copy to existing table
+CREATE TABLE cmmove3(f1 text COMPRESSION pglz);
+INSERT INTO cmmove3 SELECT * FROM cmdata;
+INSERT INTO cmmove3 SELECT * FROM cmdata1;
+SELECT pg_column_compression(f1) FROM cmmove2;
+
+-- test external compressed data
+CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS
+'select array_agg(md5(g::text))::text from generate_series(1, 256) g';
+CREATE TABLE cmdata2 (f1 text COMPRESSION pglz);
+INSERT INTO cmdata2 select large_val() || repeat('a', 4000);
+SELECT pg_column_compression(f1) FROM cmdata2;
+INSERT INTO cmdata1 SELECT * FROM cmdata2;
+SELECT pg_column_compression(f1) FROM cmdata1;
+DROP TABLE cmdata2;
+
+-- test LIKE INCLUDING COMPRESSION
+CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION);
+\d+ cmdata2
+
+-- test compression with materialized view
+CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1;
+\d+ mv
+SELECT pg_column_compression(f1) FROM cmdata1;
+SELECT pg_column_compression(x) FROM mv;
+
+-- test compression with partition
+CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1);
+CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0);
+CREATE TABLE cmpart2(f1 text COMPRESSION pglz);
+
+ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1);
+INSERT INTO cmpart VALUES (repeat('123456789',1004));
+INSERT INTO cmpart VALUES (repeat('123456789',4004));
+SELECT pg_column_compression(f1) FROM cmpart;
+
+-- test compression with inheritence, error
+CREATE TABLE cminh() INHERITS(cmdata, cmdata1);
+CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata);
+
+-- check data is ok
+SELECT length(f1) FROM cmdata;
+SELECT length(f1) FROM cmdata1;
+SELECT length(f1) FROM cmmove1;
+SELECT length(f1) FROM cmmove2;
+SELECT length(f1) FROM cmmove3;
+
+\set HIDE_COMPRESSAM on
diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm
index 2aa062b2c9..c64b369047 100644
--- a/src/tools/msvc/Solution.pm
+++ b/src/tools/msvc/Solution.pm
@@ -307,6 +307,7 @@ sub GenerateFiles
HAVE_LIBXML2 => undef,
HAVE_LIBXSLT => undef,
HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef,
+ HAVE_LIBLZ4 => undef,
HAVE_LINK => undef,
HAVE_LOCALE_T => 1,
HAVE_LONG_INT_64 => undef,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index bab4f3adb3..46cac11d39 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -395,6 +395,7 @@ CompositeIOData
CompositeTypeStmt
CompoundAffixFlag
CompressionAlgorithm
+CompressionAmRoutine
CompressorState
ComputeXidHorizonsResult
ConditionVariable
--
2.17.0
--YZ5djTAD1cGYuMQK
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v24-0002-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch"
^ permalink raw reply [nested|flat] 14+ messages in thread
* explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE))
@ 2022-01-24 17:35 Justin Pryzby <[email protected]>
0 siblings, 2 replies; 14+ messages in thread
From: Justin Pryzby @ 2022-01-24 17:35 UTC (permalink / raw)
To: Nikolay Samokhvalov <[email protected]>; +Cc: [email protected]
I'm renaming this thread for better visibility, since buffers is a small,
optional part of the patches I sent.
I made a CF entry here.
https://commitfest.postgresql.org/36/3409/
On Wed, Dec 01, 2021 at 06:58:20PM -0600, Justin Pryzby wrote:
> On Mon, Nov 15, 2021 at 01:09:54PM -0600, Justin Pryzby wrote:
> > Some time ago, I had a few relevant patches:
> > 1) add explain(REGRESS) which is shorthand for (BUFFERS OFF, TIMING OFF, COSTS OFF, SUMMARY OFF)
> > 2) add explain(MACHINE) which elides machine-specific output from explain;
> > for example, Heap Fetches, sort spaceUsed, hash nbuckets, and tidbitmap stuff.
> >
> > https://www.postgresql.org/message-id/flat/[email protected]
>
> The attached patch series now looks like this (some minor patches are not
> included in this list):
>
> 1. add GUC explain_regress, which disables TIMING, SUMMARY, COSTS;
> 2. enable explain(BUFFERS) by default (but disabled by explain_regress);
> 3. Add explain(MACHINE) - which is disabled by explain_regress.
> This elides various machine-specific output like Memory and Disk use.
> Maybe it should be called something else like "QUIET" or "VERBOSE_MINUS_1"
> or ??
>
> The regression tests now automatically run with explain_regress=on, which is
> shorthand for TIMING OFF, SUMMARY OFF, COSTS OFF, and then BUFFERS OFF.
>
> There's a further option called explain(MACHINE) which can be disabled to hide
> portions of the output that are unstable, like Memory/Disk/Batches/
> Heap Fetches/Heap Blocks. This allows "explain analyze" to be used more easily
> in regression tests, and simplifies some existing tests that currently use
> plpgsql functions to filter the output. But it doesn't handle all the
> variations from parallel workers.
>
> (3) is optional, but simplifies some regression tests. The patch series could
> be rephrased with (3) first.
>
> Unfortunately, "COSTS OFF" breaks postgres_fdw remote_estimate. If specifying
> "COSTS ON" in postgres_fdw.c is considered to be a poor fix , then I suppose
> this patch series could do as suggested and enable buffers by default only when
> ANALYZE is specified. Then postgres_fdw is not affected, and the
> explain_regress GUC is optional: instead, we'd need to specify BUFFERS OFF in
> ~100 regression tests which use EXPLAIN ANALYZE. (3) still seems useful on its
> own.
Attachments:
[text/x-diff] 0001-Add-GUC-explain_regress.patch (9.3K, ../../[email protected]/2-0001-Add-GUC-explain_regress.patch)
download | inline diff:
From 747cb7a979cf96f99403a005053e635f3bf8c3f0 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Sat, 22 Feb 2020 21:17:10 -0600
Subject: [PATCH 1/7] Add GUC: explain_regress
This changes the defaults for explain to: costs off, timing off, summary off.
It'd be reasonable to use this for new regression tests which are not intended
to be backpatched.
---
contrib/postgres_fdw/postgres_fdw.c | 2 +-
src/backend/commands/explain.c | 13 +++++++++++--
src/backend/utils/misc/guc.c | 13 +++++++++++++
src/include/commands/explain.h | 2 ++
src/test/regress/expected/explain.out | 3 +++
src/test/regress/expected/inherit.out | 2 +-
src/test/regress/expected/stats_ext.out | 2 +-
src/test/regress/pg_regress.c | 3 ++-
src/test/regress/sql/explain.sql | 4 ++++
src/test/regress/sql/inherit.sql | 2 +-
src/test/regress/sql/stats_ext.sql | 2 +-
11 files changed, 40 insertions(+), 8 deletions(-)
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index bf3f3d9e26e..71f5b145984 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -3112,7 +3112,7 @@ estimate_path_cost_size(PlannerInfo *root,
* values, so don't request params_list.
*/
initStringInfo(&sql);
- appendStringInfoString(&sql, "EXPLAIN ");
+ appendStringInfoString(&sql, "EXPLAIN (COSTS)");
deparseSelectStmtForRel(&sql, root, foreignrel, fdw_scan_tlist,
remote_conds, pathkeys,
fpextra ? fpextra->has_final_sort : false,
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index b970997c346..43e8bc78778 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -154,6 +154,7 @@ static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
+bool explain_regress = false; /* GUC */
/*
@@ -172,6 +173,7 @@ ExplainQuery(ParseState *pstate, ExplainStmt *stmt,
ListCell *lc;
bool timing_set = false;
bool summary_set = false;
+ bool costs_set = false;
/* Parse options list. */
foreach(lc, stmt->options)
@@ -183,7 +185,11 @@ ExplainQuery(ParseState *pstate, ExplainStmt *stmt,
else if (strcmp(opt->defname, "verbose") == 0)
es->verbose = defGetBoolean(opt);
else if (strcmp(opt->defname, "costs") == 0)
+ {
+ /* Need to keep track if it was explicitly set to ON */
+ costs_set = true;
es->costs = defGetBoolean(opt);
+ }
else if (strcmp(opt->defname, "buffers") == 0)
es->buffers = defGetBoolean(opt);
else if (strcmp(opt->defname, "wal") == 0)
@@ -227,13 +233,16 @@ ExplainQuery(ParseState *pstate, ExplainStmt *stmt,
parser_errposition(pstate, opt->location)));
}
+ /* if the costs option was not set explicitly, set default value */
+ es->costs = (costs_set) ? es->costs : es->costs && !explain_regress;
+
if (es->wal && !es->analyze)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("EXPLAIN option WAL requires ANALYZE")));
/* if the timing was not set explicitly, set default value */
- es->timing = (timing_set) ? es->timing : es->analyze;
+ es->timing = (timing_set) ? es->timing : es->analyze && !explain_regress;
/* check that timing is used with EXPLAIN ANALYZE */
if (es->timing && !es->analyze)
@@ -242,7 +251,7 @@ ExplainQuery(ParseState *pstate, ExplainStmt *stmt,
errmsg("EXPLAIN option TIMING requires ANALYZE")));
/* if the summary was not set explicitly, set default value */
- es->summary = (summary_set) ? es->summary : es->analyze;
+ es->summary = (summary_set) ? es->summary : es->analyze && !explain_regress;
query = castNode(Query, stmt->query);
if (IsQueryIdEnabled())
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 4c94f09c645..69f403f0c14 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -45,6 +45,7 @@
#include "catalog/pg_authid.h"
#include "catalog/storage.h"
#include "commands/async.h"
+#include "commands/explain.h"
#include "commands/prepare.h"
#include "commands/tablespace.h"
#include "commands/trigger.h"
@@ -1470,6 +1471,18 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+
+ {
+ {"explain_regress", PGC_USERSET, DEVELOPER_OPTIONS,
+ gettext_noop("Change defaults of EXPLAIN to avoid unstable output."),
+ NULL,
+ GUC_NOT_IN_SAMPLE | GUC_EXPLAIN
+ },
+ &explain_regress,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"log_parser_stats", PGC_SUSET, STATS_MONITORING,
gettext_noop("Writes parser performance statistics to the server log."),
diff --git a/src/include/commands/explain.h b/src/include/commands/explain.h
index 666977fb1f8..ce8c458d731 100644
--- a/src/include/commands/explain.h
+++ b/src/include/commands/explain.h
@@ -61,6 +61,8 @@ typedef struct ExplainState
ExplainWorkersState *workers_state; /* needed if parallel plan */
} ExplainState;
+extern bool explain_regress;
+
/* Hook for plugins to get control in ExplainOneQuery() */
typedef void (*ExplainOneQuery_hook_type) (Query *query,
int cursorOptions,
diff --git a/src/test/regress/expected/explain.out b/src/test/regress/expected/explain.out
index 1734dfee8cc..188dc7ccec9 100644
--- a/src/test/regress/expected/explain.out
+++ b/src/test/regress/expected/explain.out
@@ -8,6 +8,9 @@
-- To produce stable regression test output, it's usually necessary to
-- ignore details such as exact costs or row counts. These filter
-- functions replace changeable output details with fixed strings.
+-- Output normal, user-facing details, not the sanitized version used for the
+-- rest of the regression tests
+set explain_regress = off;
create function explain_filter(text) returns setof text
language plpgsql as
$$
diff --git a/src/test/regress/expected/inherit.out b/src/test/regress/expected/inherit.out
index 2d49e765de8..38fb5f94c6a 100644
--- a/src/test/regress/expected/inherit.out
+++ b/src/test/regress/expected/inherit.out
@@ -664,7 +664,7 @@ select tableoid::regclass::text as relname, parted_tab.* from parted_tab order b
(3 rows)
-- modifies partition key, but no rows will actually be updated
-explain update parted_tab set a = 2 where false;
+explain (costs on) update parted_tab set a = 2 where false;
QUERY PLAN
--------------------------------------------------------
Update on parted_tab (cost=0.00..0.00 rows=0 width=0)
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 042316aeed8..638abaaf804 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -14,7 +14,7 @@ declare
first_row bool := true;
begin
for ln in
- execute format('explain analyze %s', $1)
+ execute format('explain (analyze, costs on) %s', $1)
loop
if first_row then
first_row := false;
diff --git a/src/test/regress/pg_regress.c b/src/test/regress/pg_regress.c
index e6f71c7582e..919ede8b768 100644
--- a/src/test/regress/pg_regress.c
+++ b/src/test/regress/pg_regress.c
@@ -634,7 +634,7 @@ initialize_environment(void)
* user's ability to set other variables through that.
*/
{
- const char *my_pgoptions = "-c intervalstyle=postgres_verbose";
+ const char *my_pgoptions = "-c intervalstyle=postgres_verbose -c explain_regress=on";
const char *old_pgoptions = getenv("PGOPTIONS");
char *new_pgoptions;
@@ -2296,6 +2296,7 @@ regression_main(int argc, char *argv[],
fputs("log_lock_waits = on\n", pg_conf);
fputs("log_temp_files = 128kB\n", pg_conf);
fputs("max_prepared_transactions = 2\n", pg_conf);
+ // fputs("explain_regress = yes\n", pg_conf);
for (sl = temp_configs; sl != NULL; sl = sl->next)
{
diff --git a/src/test/regress/sql/explain.sql b/src/test/regress/sql/explain.sql
index 84549c78fa8..f91d1004d5e 100644
--- a/src/test/regress/sql/explain.sql
+++ b/src/test/regress/sql/explain.sql
@@ -10,6 +10,10 @@
-- ignore details such as exact costs or row counts. These filter
-- functions replace changeable output details with fixed strings.
+-- Output normal, user-facing details, not the sanitized version used for the
+-- rest of the regression tests
+set explain_regress = off;
+
create function explain_filter(text) returns setof text
language plpgsql as
$$
diff --git a/src/test/regress/sql/inherit.sql b/src/test/regress/sql/inherit.sql
index 195aedb5ff5..868ee58b803 100644
--- a/src/test/regress/sql/inherit.sql
+++ b/src/test/regress/sql/inherit.sql
@@ -169,7 +169,7 @@ where parted_tab.a = ss.a;
select tableoid::regclass::text as relname, parted_tab.* from parted_tab order by 1,2;
-- modifies partition key, but no rows will actually be updated
-explain update parted_tab set a = 2 where false;
+explain (costs on) update parted_tab set a = 2 where false;
drop table parted_tab;
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 6b954c9e500..4c91f608634 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -16,7 +16,7 @@ declare
first_row bool := true;
begin
for ln in
- execute format('explain analyze %s', $1)
+ execute format('explain (analyze, costs on) %s', $1)
loop
if first_row then
first_row := false;
--
2.17.1
[text/x-diff] 0002-exercise-explain_regress.patch (12.8K, ../../[email protected]/3-0002-exercise-explain_regress.patch)
download | inline diff:
From 1b5d8a68ecded561bc41b52759896df298faf990 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Mon, 15 Nov 2021 21:54:12 -0600
Subject: [PATCH 2/7] exercise explain_regress
not intended to be merged, since it creates backpatch hazards (unless the GUC
is also backpatched)
---
src/test/regress/expected/matview.out | 12 ++++++------
src/test/regress/expected/select_into.out | 20 ++++++++++----------
src/test/regress/expected/tidscan.out | 6 +++---
src/test/regress/sql/matview.sql | 12 ++++++------
src/test/regress/sql/select_into.sql | 20 ++++++++++----------
src/test/regress/sql/tidscan.sql | 6 +++---
6 files changed, 38 insertions(+), 38 deletions(-)
diff --git a/src/test/regress/expected/matview.out b/src/test/regress/expected/matview.out
index 313c72a2685..e7ce5bb1887 100644
--- a/src/test/regress/expected/matview.out
+++ b/src/test/regress/expected/matview.out
@@ -606,7 +606,7 @@ GRANT ALL ON SCHEMA matview_schema TO public;
SET SESSION AUTHORIZATION regress_matview_user;
CREATE MATERIALIZED VIEW matview_schema.mv_withdata1 (a) AS
SELECT generate_series(1, 10) WITH DATA;
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE MATERIALIZED VIEW matview_schema.mv_withdata2 (a) AS
SELECT generate_series(1, 10) WITH DATA;
QUERY PLAN
@@ -618,7 +618,7 @@ EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
REFRESH MATERIALIZED VIEW matview_schema.mv_withdata2;
CREATE MATERIALIZED VIEW matview_schema.mv_nodata1 (a) AS
SELECT generate_series(1, 10) WITH NO DATA;
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE MATERIALIZED VIEW matview_schema.mv_nodata2 (a) AS
SELECT generate_series(1, 10) WITH NO DATA;
QUERY PLAN
@@ -651,11 +651,11 @@ ERROR: relation "matview_ine_tab" already exists
CREATE MATERIALIZED VIEW IF NOT EXISTS matview_ine_tab AS
SELECT 1 / 0 WITH NO DATA; -- ok
NOTICE: relation "matview_ine_tab" already exists, skipping
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE MATERIALIZED VIEW matview_ine_tab AS
SELECT 1 / 0; -- error
ERROR: relation "matview_ine_tab" already exists
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE MATERIALIZED VIEW IF NOT EXISTS matview_ine_tab AS
SELECT 1 / 0; -- ok
NOTICE: relation "matview_ine_tab" already exists, skipping
@@ -663,11 +663,11 @@ NOTICE: relation "matview_ine_tab" already exists, skipping
------------
(0 rows)
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE MATERIALIZED VIEW matview_ine_tab AS
SELECT 1 / 0 WITH NO DATA; -- error
ERROR: relation "matview_ine_tab" already exists
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE MATERIALIZED VIEW IF NOT EXISTS matview_ine_tab AS
SELECT 1 / 0 WITH NO DATA; -- ok
NOTICE: relation "matview_ine_tab" already exists, skipping
diff --git a/src/test/regress/expected/select_into.out b/src/test/regress/expected/select_into.out
index 43b8209d229..3de0be2a154 100644
--- a/src/test/regress/expected/select_into.out
+++ b/src/test/regress/expected/select_into.out
@@ -25,7 +25,7 @@ CREATE TABLE selinto_schema.tbl_withdata1 (a)
AS SELECT generate_series(1,3) WITH DATA;
INSERT INTO selinto_schema.tbl_withdata1 VALUES (4);
ERROR: permission denied for table tbl_withdata1
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE selinto_schema.tbl_withdata2 (a) AS
SELECT generate_series(1,3) WITH DATA;
QUERY PLAN
@@ -37,7 +37,7 @@ EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
-- WITH NO DATA, passes.
CREATE TABLE selinto_schema.tbl_nodata1 (a) AS
SELECT generate_series(1,3) WITH NO DATA;
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE selinto_schema.tbl_nodata2 (a) AS
SELECT generate_series(1,3) WITH NO DATA;
QUERY PLAN
@@ -50,7 +50,7 @@ EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
PREPARE data_sel AS SELECT generate_series(1,3);
CREATE TABLE selinto_schema.tbl_withdata3 (a) AS
EXECUTE data_sel WITH DATA;
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE selinto_schema.tbl_withdata4 (a) AS
EXECUTE data_sel WITH DATA;
QUERY PLAN
@@ -62,7 +62,7 @@ EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
-- EXECUTE and WITH NO DATA, passes.
CREATE TABLE selinto_schema.tbl_nodata3 (a) AS
EXECUTE data_sel WITH NO DATA;
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE selinto_schema.tbl_nodata4 (a) AS
EXECUTE data_sel WITH NO DATA;
QUERY PLAN
@@ -188,20 +188,20 @@ CREATE TABLE ctas_ine_tbl AS SELECT 1 / 0 WITH NO DATA; -- error
ERROR: relation "ctas_ine_tbl" already exists
CREATE TABLE IF NOT EXISTS ctas_ine_tbl AS SELECT 1 / 0 WITH NO DATA; -- ok
NOTICE: relation "ctas_ine_tbl" already exists, skipping
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE ctas_ine_tbl AS SELECT 1 / 0; -- error
ERROR: relation "ctas_ine_tbl" already exists
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE IF NOT EXISTS ctas_ine_tbl AS SELECT 1 / 0; -- ok
NOTICE: relation "ctas_ine_tbl" already exists, skipping
QUERY PLAN
------------
(0 rows)
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE ctas_ine_tbl AS SELECT 1 / 0 WITH NO DATA; -- error
ERROR: relation "ctas_ine_tbl" already exists
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE IF NOT EXISTS ctas_ine_tbl AS SELECT 1 / 0 WITH NO DATA; -- ok
NOTICE: relation "ctas_ine_tbl" already exists, skipping
QUERY PLAN
@@ -209,10 +209,10 @@ NOTICE: relation "ctas_ine_tbl" already exists, skipping
(0 rows)
PREPARE ctas_ine_query AS SELECT 1 / 0;
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE ctas_ine_tbl AS EXECUTE ctas_ine_query; -- error
ERROR: relation "ctas_ine_tbl" already exists
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE IF NOT EXISTS ctas_ine_tbl AS EXECUTE ctas_ine_query; -- ok
NOTICE: relation "ctas_ine_tbl" already exists, skipping
QUERY PLAN
diff --git a/src/test/regress/expected/tidscan.out b/src/test/regress/expected/tidscan.out
index 13c3c360c25..de93145bf0d 100644
--- a/src/test/regress/expected/tidscan.out
+++ b/src/test/regress/expected/tidscan.out
@@ -189,7 +189,7 @@ FETCH NEXT FROM c;
(1 row)
-- perform update
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
UPDATE tidscan SET id = -id WHERE CURRENT OF c RETURNING *;
QUERY PLAN
---------------------------------------------------
@@ -205,7 +205,7 @@ FETCH NEXT FROM c;
(1 row)
-- perform update
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
UPDATE tidscan SET id = -id WHERE CURRENT OF c RETURNING *;
QUERY PLAN
---------------------------------------------------
@@ -229,7 +229,7 @@ FETCH NEXT FROM c;
(0 rows)
-- should error out
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
UPDATE tidscan SET id = -id WHERE CURRENT OF c RETURNING *;
ERROR: cursor "c" is not positioned on a row
ROLLBACK;
diff --git a/src/test/regress/sql/matview.sql b/src/test/regress/sql/matview.sql
index 68b9ccfd452..e0b562933d0 100644
--- a/src/test/regress/sql/matview.sql
+++ b/src/test/regress/sql/matview.sql
@@ -255,13 +255,13 @@ GRANT ALL ON SCHEMA matview_schema TO public;
SET SESSION AUTHORIZATION regress_matview_user;
CREATE MATERIALIZED VIEW matview_schema.mv_withdata1 (a) AS
SELECT generate_series(1, 10) WITH DATA;
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE MATERIALIZED VIEW matview_schema.mv_withdata2 (a) AS
SELECT generate_series(1, 10) WITH DATA;
REFRESH MATERIALIZED VIEW matview_schema.mv_withdata2;
CREATE MATERIALIZED VIEW matview_schema.mv_nodata1 (a) AS
SELECT generate_series(1, 10) WITH NO DATA;
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE MATERIALIZED VIEW matview_schema.mv_nodata2 (a) AS
SELECT generate_series(1, 10) WITH NO DATA;
REFRESH MATERIALIZED VIEW matview_schema.mv_nodata2;
@@ -282,16 +282,16 @@ CREATE MATERIALIZED VIEW matview_ine_tab AS
SELECT 1 / 0 WITH NO DATA; -- error
CREATE MATERIALIZED VIEW IF NOT EXISTS matview_ine_tab AS
SELECT 1 / 0 WITH NO DATA; -- ok
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE MATERIALIZED VIEW matview_ine_tab AS
SELECT 1 / 0; -- error
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE MATERIALIZED VIEW IF NOT EXISTS matview_ine_tab AS
SELECT 1 / 0; -- ok
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE MATERIALIZED VIEW matview_ine_tab AS
SELECT 1 / 0 WITH NO DATA; -- error
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE MATERIALIZED VIEW IF NOT EXISTS matview_ine_tab AS
SELECT 1 / 0 WITH NO DATA; -- ok
DROP MATERIALIZED VIEW matview_ine_tab;
diff --git a/src/test/regress/sql/select_into.sql b/src/test/regress/sql/select_into.sql
index 7e903c339a8..7031e86a2c9 100644
--- a/src/test/regress/sql/select_into.sql
+++ b/src/test/regress/sql/select_into.sql
@@ -30,26 +30,26 @@ SET SESSION AUTHORIZATION regress_selinto_user;
CREATE TABLE selinto_schema.tbl_withdata1 (a)
AS SELECT generate_series(1,3) WITH DATA;
INSERT INTO selinto_schema.tbl_withdata1 VALUES (4);
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE selinto_schema.tbl_withdata2 (a) AS
SELECT generate_series(1,3) WITH DATA;
-- WITH NO DATA, passes.
CREATE TABLE selinto_schema.tbl_nodata1 (a) AS
SELECT generate_series(1,3) WITH NO DATA;
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE selinto_schema.tbl_nodata2 (a) AS
SELECT generate_series(1,3) WITH NO DATA;
-- EXECUTE and WITH DATA, passes.
PREPARE data_sel AS SELECT generate_series(1,3);
CREATE TABLE selinto_schema.tbl_withdata3 (a) AS
EXECUTE data_sel WITH DATA;
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE selinto_schema.tbl_withdata4 (a) AS
EXECUTE data_sel WITH DATA;
-- EXECUTE and WITH NO DATA, passes.
CREATE TABLE selinto_schema.tbl_nodata3 (a) AS
EXECUTE data_sel WITH NO DATA;
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE selinto_schema.tbl_nodata4 (a) AS
EXECUTE data_sel WITH NO DATA;
RESET SESSION AUTHORIZATION;
@@ -122,17 +122,17 @@ CREATE TABLE ctas_ine_tbl AS SELECT 1 / 0; -- error
CREATE TABLE IF NOT EXISTS ctas_ine_tbl AS SELECT 1 / 0; -- ok
CREATE TABLE ctas_ine_tbl AS SELECT 1 / 0 WITH NO DATA; -- error
CREATE TABLE IF NOT EXISTS ctas_ine_tbl AS SELECT 1 / 0 WITH NO DATA; -- ok
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE ctas_ine_tbl AS SELECT 1 / 0; -- error
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE IF NOT EXISTS ctas_ine_tbl AS SELECT 1 / 0; -- ok
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE ctas_ine_tbl AS SELECT 1 / 0 WITH NO DATA; -- error
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE IF NOT EXISTS ctas_ine_tbl AS SELECT 1 / 0 WITH NO DATA; -- ok
PREPARE ctas_ine_query AS SELECT 1 / 0;
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE ctas_ine_tbl AS EXECUTE ctas_ine_query; -- error
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE IF NOT EXISTS ctas_ine_tbl AS EXECUTE ctas_ine_query; -- ok
DROP TABLE ctas_ine_tbl;
diff --git a/src/test/regress/sql/tidscan.sql b/src/test/regress/sql/tidscan.sql
index 313e0fb9b67..3d1f447088f 100644
--- a/src/test/regress/sql/tidscan.sql
+++ b/src/test/regress/sql/tidscan.sql
@@ -68,17 +68,17 @@ DECLARE c CURSOR FOR SELECT ctid, * FROM tidscan;
FETCH NEXT FROM c; -- skip one row
FETCH NEXT FROM c;
-- perform update
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
UPDATE tidscan SET id = -id WHERE CURRENT OF c RETURNING *;
FETCH NEXT FROM c;
-- perform update
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
UPDATE tidscan SET id = -id WHERE CURRENT OF c RETURNING *;
SELECT * FROM tidscan;
-- position cursor past any rows
FETCH NEXT FROM c;
-- should error out
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
UPDATE tidscan SET id = -id WHERE CURRENT OF c RETURNING *;
ROLLBACK;
--
2.17.1
[text/x-diff] 0003-Make-explain-default-to-BUFFERS-TRUE.patch (4.7K, ../../[email protected]/4-0003-Make-explain-default-to-BUFFERS-TRUE.patch)
download | inline diff:
From c176127e9da57fb85c93a541f9c110b1551c5b98 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Wed, 22 Jul 2020 19:20:40 -0500
Subject: [PATCH 3/7] Make explain default to BUFFERS TRUE
---
contrib/auto_explain/auto_explain.c | 4 ++--
doc/src/sgml/config.sgml | 2 +-
doc/src/sgml/perform.sgml | 4 ++--
doc/src/sgml/ref/explain.sgml | 2 +-
src/backend/commands/explain.c | 8 ++++++++
5 files changed, 14 insertions(+), 6 deletions(-)
diff --git a/contrib/auto_explain/auto_explain.c b/contrib/auto_explain/auto_explain.c
index 3e09abaecac..1ed43d3c8a8 100644
--- a/contrib/auto_explain/auto_explain.c
+++ b/contrib/auto_explain/auto_explain.c
@@ -27,7 +27,7 @@ PG_MODULE_MAGIC;
static int auto_explain_log_min_duration = -1; /* msec or -1 */
static bool auto_explain_log_analyze = false;
static bool auto_explain_log_verbose = false;
-static bool auto_explain_log_buffers = false;
+static bool auto_explain_log_buffers = true;
static bool auto_explain_log_wal = false;
static bool auto_explain_log_triggers = false;
static bool auto_explain_log_timing = true;
@@ -143,7 +143,7 @@ _PG_init(void)
"Log buffers usage.",
NULL,
&auto_explain_log_buffers,
- false,
+ true,
PGC_SUSET,
0,
NULL,
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 4cd9818acf8..a99bd58f8ba 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -7813,7 +7813,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
displayed in <link linkend="monitoring-pg-stat-database-view">
<structname>pg_stat_database</structname></link>, in the output of
<xref linkend="sql-explain"/> when the <literal>BUFFERS</literal> option
- is used, in the output of <xref linkend="sql-vacuum"/> when
+ is enabled, in the output of <xref linkend="sql-vacuum"/> when
the <literal>VERBOSE</literal> option is used, by autovacuum
for auto-vacuums and auto-analyzes, when <xref
linkend="guc-log-autovacuum-min-duration"/> is set and by
diff --git a/doc/src/sgml/perform.sgml b/doc/src/sgml/perform.sgml
index 89ff58338e5..fc4367ffe6e 100644
--- a/doc/src/sgml/perform.sgml
+++ b/doc/src/sgml/perform.sgml
@@ -731,8 +731,8 @@ EXPLAIN ANALYZE SELECT * FROM polygon_tbl WHERE f1 @> polygon '(0.5,2.0)';
</para>
<para>
- <command>EXPLAIN</command> has a <literal>BUFFERS</literal> option that can be used with
- <literal>ANALYZE</literal> to get even more run time statistics:
+ <command>EXPLAIN ANALYZE</command> has a <literal>BUFFERS</literal> option which
+ provides even more run time statistics:
<screen>
EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM tenk1 WHERE unique1 < 100 AND unique2 > 9000;
diff --git a/doc/src/sgml/ref/explain.sgml b/doc/src/sgml/ref/explain.sgml
index 4d758fb237e..ceb0f4c83a1 100644
--- a/doc/src/sgml/ref/explain.sgml
+++ b/doc/src/sgml/ref/explain.sgml
@@ -190,7 +190,7 @@ ROLLBACK;
The number of blocks shown for an
upper-level node includes those used by all its child nodes. In text
format, only non-zero values are printed. It defaults to
- <literal>FALSE</literal>.
+ <literal>TRUE</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index 43e8bc78778..6cc04f90fb4 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -174,6 +174,7 @@ ExplainQuery(ParseState *pstate, ExplainStmt *stmt,
bool timing_set = false;
bool summary_set = false;
bool costs_set = false;
+ bool buffers_set = false;
/* Parse options list. */
foreach(lc, stmt->options)
@@ -191,7 +192,10 @@ ExplainQuery(ParseState *pstate, ExplainStmt *stmt,
es->costs = defGetBoolean(opt);
}
else if (strcmp(opt->defname, "buffers") == 0)
+ {
+ buffers_set = true;
es->buffers = defGetBoolean(opt);
+ }
else if (strcmp(opt->defname, "wal") == 0)
es->wal = defGetBoolean(opt);
else if (strcmp(opt->defname, "settings") == 0)
@@ -253,6 +257,9 @@ ExplainQuery(ParseState *pstate, ExplainStmt *stmt,
/* if the summary was not set explicitly, set default value */
es->summary = (summary_set) ? es->summary : es->analyze && !explain_regress;
+ /* if the buffers option was not set explicitly, set default value */
+ es->buffers = (buffers_set) ? es->buffers : !explain_regress;
+
query = castNode(Query, stmt->query);
if (IsQueryIdEnabled())
jstate = JumbleQuery(query, pstate->p_sourcetext);
@@ -323,6 +330,7 @@ NewExplainState(void)
/* Set default options (most fields can be left as zeroes). */
es->costs = true;
+ es->buffers = true;
/* Prepare output buffer. */
es->str = makeStringInfo();
--
2.17.1
[text/x-diff] 0004-Add-explain-MACHINE-to-hide-machine-dependent-output.patch (33.9K, ../../[email protected]/5-0004-Add-explain-MACHINE-to-hide-machine-dependent-output.patch)
download | inline diff:
From a02bd2897add538803a72d20a7fd9030f9b406d3 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Sat, 22 Feb 2020 18:45:22 -0600
Subject: [PATCH 4/7] Add explain(MACHINE) to hide machine-dependent output..
This new option hides some output that has traditionally been shown; the option
is enabled by regression mode to hide unstable output.
This allows EXPLAIN ANALYZE to be used in regression tests with stable output.
This is like a "quiet" mode, or negative verbosity.
Also add regression tests for HashAgg and Bitmap scan, which previously had no
tests with explain(analyze).
This does not handle variations in "Workers Launched", or other parallel worker
bits which are handled by force_parallel_mode=regress.
Also add tests for show_hashagg_info and tidbitmap, for which there's no other
test.
---
src/backend/commands/explain.c | 77 +++++++++++++------
src/include/commands/explain.h | 1 +
src/test/isolation/expected/horizons.out | 40 +++++-----
src/test/isolation/specs/horizons.spec | 2 +-
src/test/regress/expected/explain.out | 55 +++++++++++++
.../regress/expected/incremental_sort.out | 4 +-
src/test/regress/expected/memoize.out | 35 ++++-----
src/test/regress/expected/partition_prune.out | 4 +-
src/test/regress/expected/select_parallel.out | 32 +++-----
src/test/regress/expected/subselect.out | 21 +----
src/test/regress/sql/explain.sql | 17 ++++
src/test/regress/sql/memoize.sql | 4 +-
src/test/regress/sql/select_parallel.sql | 20 +----
src/test/regress/sql/subselect.sql | 19 +----
14 files changed, 182 insertions(+), 149 deletions(-)
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index 6cc04f90fb4..8b4ff9624d4 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -175,6 +175,7 @@ ExplainQuery(ParseState *pstate, ExplainStmt *stmt,
bool summary_set = false;
bool costs_set = false;
bool buffers_set = false;
+ bool machine_set = false;
/* Parse options list. */
foreach(lc, stmt->options)
@@ -210,6 +211,11 @@ ExplainQuery(ParseState *pstate, ExplainStmt *stmt,
summary_set = true;
es->summary = defGetBoolean(opt);
}
+ else if (strcmp(opt->defname, "machine") == 0)
+ {
+ machine_set = true;
+ es->machine = defGetBoolean(opt);
+ }
else if (strcmp(opt->defname, "format") == 0)
{
char *p = defGetString(opt);
@@ -260,6 +266,9 @@ ExplainQuery(ParseState *pstate, ExplainStmt *stmt,
/* if the buffers option was not set explicitly, set default value */
es->buffers = (buffers_set) ? es->buffers : !explain_regress;
+ /* if the machine option was not set explicitly, set default value */
+ es->machine = (machine_set) ? es->machine : !explain_regress;
+
query = castNode(Query, stmt->query);
if (IsQueryIdEnabled())
jstate = JumbleQuery(query, pstate->p_sourcetext);
@@ -621,7 +630,7 @@ ExplainOnePlan(PlannedStmt *plannedstmt, IntoClause *into, ExplainState *es,
/* Create textual dump of plan tree */
ExplainPrintPlan(es, queryDesc);
- if (es->verbose && plannedstmt->queryId != UINT64CONST(0))
+ if (es->verbose && plannedstmt->queryId != UINT64CONST(0) && es->machine)
{
/*
* Output the queryid as an int64 rather than a uint64 so we match
@@ -632,7 +641,7 @@ ExplainOnePlan(PlannedStmt *plannedstmt, IntoClause *into, ExplainState *es,
}
/* Show buffer usage in planning */
- if (bufusage)
+ if (bufusage && es->buffers)
{
ExplainOpenGroup("Planning", "Planning", true, es);
show_buffer_usage(es, bufusage, true);
@@ -1772,7 +1781,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
if (plan->qual)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
- if (es->analyze)
+ if (es->analyze && es->machine)
ExplainPropertyFloat("Heap Fetches", NULL,
planstate->instrument->ntuples2, 0, es);
break;
@@ -2746,8 +2755,12 @@ show_sort_info(SortState *sortstate, ExplainState *es)
if (es->format == EXPLAIN_FORMAT_TEXT)
{
ExplainIndentText(es);
- appendStringInfo(es->str, "Sort Method: %s %s: " INT64_FORMAT "kB\n",
- sortMethod, spaceType, spaceUsed);
+ appendStringInfo(es->str, "Sort Method: %s",
+ sortMethod);
+ if (es->machine)
+ appendStringInfo(es->str, " %s: " INT64_FORMAT "kB",
+ spaceType, spaceUsed);
+ appendStringInfoString(es->str, "\n");
}
else
{
@@ -2791,8 +2804,12 @@ show_sort_info(SortState *sortstate, ExplainState *es)
{
ExplainIndentText(es);
appendStringInfo(es->str,
- "Sort Method: %s %s: " INT64_FORMAT "kB\n",
- sortMethod, spaceType, spaceUsed);
+ "Sort Method: %s",
+ sortMethod);
+ if (es->machine)
+ appendStringInfo(es->str, " %s: " INT64_FORMAT "kB", spaceType, spaceUsed);
+
+ appendStringInfoString(es->str, "\n");
}
else
{
@@ -3080,25 +3097,26 @@ show_hash_info(HashState *hashstate, ExplainState *es)
ExplainPropertyInteger("Peak Memory Usage", "kB",
spacePeakKb, es);
}
- else if (hinstrument.nbatch_original != hinstrument.nbatch ||
- hinstrument.nbuckets_original != hinstrument.nbuckets)
+ else
{
ExplainIndentText(es);
- appendStringInfo(es->str,
- "Buckets: %d (originally %d) Batches: %d (originally %d) Memory Usage: %ldkB\n",
+ if (hinstrument.nbatch_original != hinstrument.nbatch ||
+ hinstrument.nbuckets_original != hinstrument.nbuckets)
+ appendStringInfo(es->str,
+ "Buckets: %d (originally %d) Batches: %d (originally %d)",
hinstrument.nbuckets,
hinstrument.nbuckets_original,
hinstrument.nbatch,
- hinstrument.nbatch_original,
- spacePeakKb);
- }
- else
- {
- ExplainIndentText(es);
- appendStringInfo(es->str,
- "Buckets: %d Batches: %d Memory Usage: %ldkB\n",
- hinstrument.nbuckets, hinstrument.nbatch,
- spacePeakKb);
+ hinstrument.nbatch_original);
+ else
+ appendStringInfo(es->str,
+ "Buckets: %d Batches: %d",
+ hinstrument.nbuckets, hinstrument.nbatch);
+
+ if (es->machine)
+ appendStringInfo(es->str, " Memory Usage: %ldkB", spacePeakKb);
+
+ appendStringInfoChar(es->str, '\n');
}
}
}
@@ -3182,12 +3200,16 @@ show_memoize_info(MemoizeState *mstate, List *ancestors, ExplainState *es)
{
ExplainIndentText(es);
appendStringInfo(es->str,
- "Hits: " UINT64_FORMAT " Misses: " UINT64_FORMAT " Evictions: " UINT64_FORMAT " Overflows: " UINT64_FORMAT " Memory Usage: " INT64_FORMAT "kB\n",
+ "Hits: " UINT64_FORMAT " Misses: " UINT64_FORMAT " Evictions: " UINT64_FORMAT " Overflows: " UINT64_FORMAT,
mstate->stats.cache_hits,
mstate->stats.cache_misses,
mstate->stats.cache_evictions,
- mstate->stats.cache_overflows,
+ mstate->stats.cache_overflows);
+ if (es->machine)
+ appendStringInfo(es->str, " Memory Usage: " INT64_FORMAT "kB",
memPeakKb);
+
+ appendStringInfoChar(es->str, '\n');
}
}
@@ -3256,13 +3278,16 @@ show_hashagg_info(AggState *aggstate, ExplainState *es)
Agg *agg = (Agg *) aggstate->ss.ps.plan;
int64 memPeakKb = (aggstate->hash_mem_peak + 1023) / 1024;
+ /* XXX: there's nothing portable we can show here ? */
+ if (!es->machine)
+ return;
+
if (agg->aggstrategy != AGG_HASHED &&
agg->aggstrategy != AGG_MIXED)
return;
if (es->format != EXPLAIN_FORMAT_TEXT)
{
-
if (es->costs)
ExplainPropertyInteger("Planned Partitions", NULL,
aggstate->hash_planned_partitions, es);
@@ -3375,6 +3400,10 @@ show_hashagg_info(AggState *aggstate, ExplainState *es)
static void
show_tidbitmap_info(BitmapHeapScanState *planstate, ExplainState *es)
{
+ /* XXX: there's nothing portable we can show here ? */
+ if (!es->machine)
+ return;
+
if (es->format != EXPLAIN_FORMAT_TEXT)
{
ExplainPropertyInteger("Exact Heap Blocks", NULL,
diff --git a/src/include/commands/explain.h b/src/include/commands/explain.h
index ce8c458d731..931dad08cbc 100644
--- a/src/include/commands/explain.h
+++ b/src/include/commands/explain.h
@@ -46,6 +46,7 @@ typedef struct ExplainState
bool timing; /* print detailed node timing */
bool summary; /* print total planning and execution timing */
bool settings; /* print modified settings */
+ bool machine; /* print memory/disk and other machine-specific output */
ExplainFormat format; /* output format */
/* state for output formatting --- not reset for each new plan tree */
int indent; /* current indentation level */
diff --git a/src/test/isolation/expected/horizons.out b/src/test/isolation/expected/horizons.out
index 4150b2dee64..ee3e495a646 100644
--- a/src/test/isolation/expected/horizons.out
+++ b/src/test/isolation/expected/horizons.out
@@ -24,7 +24,7 @@ Index Only Scan using horizons_tst_data_key on horizons_tst
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -34,7 +34,7 @@ step pruner_query:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -47,7 +47,7 @@ step pruner_delete:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -57,7 +57,7 @@ step pruner_query:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -94,7 +94,7 @@ Index Only Scan using horizons_tst_data_key on horizons_tst
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -104,7 +104,7 @@ step pruner_query:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -117,7 +117,7 @@ step pruner_delete:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -127,7 +127,7 @@ step pruner_query:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -156,7 +156,7 @@ step ll_start:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -166,7 +166,7 @@ step pruner_query:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -180,7 +180,7 @@ step pruner_delete:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -190,7 +190,7 @@ step pruner_query:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -220,7 +220,7 @@ step ll_start:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -230,7 +230,7 @@ step pruner_query:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -246,7 +246,7 @@ step pruner_vacuum:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -256,7 +256,7 @@ step pruner_query:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -285,7 +285,7 @@ step ll_start:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -295,7 +295,7 @@ step pruner_query:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -311,7 +311,7 @@ step pruner_vacuum:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -321,7 +321,7 @@ step pruner_query:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
diff --git a/src/test/isolation/specs/horizons.spec b/src/test/isolation/specs/horizons.spec
index d5239ff2287..082205d36ba 100644
--- a/src/test/isolation/specs/horizons.spec
+++ b/src/test/isolation/specs/horizons.spec
@@ -82,7 +82,7 @@ step pruner_vacuum
step pruner_query
{
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
}
diff --git a/src/test/regress/expected/explain.out b/src/test/regress/expected/explain.out
index 188dc7ccec9..36c20e06809 100644
--- a/src/test/regress/expected/explain.out
+++ b/src/test/regress/expected/explain.out
@@ -276,6 +276,61 @@ select explain_filter('explain (buffers, format json) select * from int8_tbl i8'
]
(1 row)
+-- HashAgg
+begin;
+SET work_mem='64kB';
+select explain_filter('explain (analyze) SELECT a, COUNT(1) FROM generate_series(1,9999)a GROUP BY 1');
+ explain_filter
+----------------------------------------------------------------------------------------------------------------
+ HashAggregate (cost=N.N..N.N rows=N width=N) (actual time=N.N..N.N rows=N loops=N)
+ Group Key: a
+ Batches: N Memory Usage: NkB Disk Usage: NkB
+ -> Function Scan on generate_series a (cost=N.N..N.N rows=N width=N) (actual time=N.N..N.N rows=N loops=N)
+ Planning Time: N.N ms
+ Execution Time: N.N ms
+(6 rows)
+
+select explain_filter('explain (analyze, machine off) SELECT a, COUNT(1) FROM generate_series(1,9999)a GROUP BY 1');
+ explain_filter
+----------------------------------------------------------------------------------------------------------------
+ HashAggregate (cost=N.N..N.N rows=N width=N) (actual time=N.N..N.N rows=N loops=N)
+ Group Key: a
+ -> Function Scan on generate_series a (cost=N.N..N.N rows=N width=N) (actual time=N.N..N.N rows=N loops=N)
+ Planning Time: N.N ms
+ Execution Time: N.N ms
+(5 rows)
+
+rollback;
+-- Bitmap scan
+begin;
+SET enable_indexscan=no;
+CREATE TABLE explainbitmap AS SELECT i AS a FROM generate_series(1,999) AS i;
+ANALYZE explainbitmap;
+CREATE INDEX ON explainbitmap(a);
+select explain_filter('explain (analyze) SELECT * FROM explainbitmap WHERE a<9');
+ explain_filter
+----------------------------------------------------------------------------------------------------------------------
+ Bitmap Heap Scan on explainbitmap (cost=N.N..N.N rows=N width=N) (actual time=N.N..N.N rows=N loops=N)
+ Recheck Cond: (a < N)
+ Heap Blocks: exact=N
+ -> Bitmap Index Scan on explainbitmap_a_idx (cost=N.N..N.N rows=N width=N) (actual time=N.N..N.N rows=N loops=N)
+ Index Cond: (a < N)
+ Planning Time: N.N ms
+ Execution Time: N.N ms
+(7 rows)
+
+select explain_filter('explain (analyze, machine off) SELECT * FROM explainbitmap WHERE a<9');
+ explain_filter
+----------------------------------------------------------------------------------------------------------------------
+ Bitmap Heap Scan on explainbitmap (cost=N.N..N.N rows=N width=N) (actual time=N.N..N.N rows=N loops=N)
+ Recheck Cond: (a < N)
+ -> Bitmap Index Scan on explainbitmap_a_idx (cost=N.N..N.N rows=N width=N) (actual time=N.N..N.N rows=N loops=N)
+ Index Cond: (a < N)
+ Planning Time: N.N ms
+ Execution Time: N.N ms
+(6 rows)
+
+rollback;
-- SETTINGS option
-- We have to ignore other settings that might be imposed by the environment,
-- so printing the whole Settings field unfortunately won't do.
diff --git a/src/test/regress/expected/incremental_sort.out b/src/test/regress/expected/incremental_sort.out
index 545e301e482..56a32df5369 100644
--- a/src/test/regress/expected/incremental_sort.out
+++ b/src/test/regress/expected/incremental_sort.out
@@ -542,7 +542,7 @@ select explain_analyze_without_memory('select * from (select * from t order by a
Full-sort Groups: 2 Sort Methods: top-N heapsort, quicksort Average Memory: NNkB Peak Memory: NNkB
-> Sort (actual rows=101 loops=1)
Sort Key: t.a
- Sort Method: quicksort Memory: NNkB
+ Sort Method: quicksort
-> Seq Scan on t (actual rows=1000 loops=1)
(9 rows)
@@ -745,7 +745,7 @@ select explain_analyze_without_memory('select * from (select * from t order by a
Pre-sorted Groups: 5 Sort Methods: top-N heapsort, quicksort Average Memory: NNkB Peak Memory: NNkB
-> Sort (actual rows=1000 loops=1)
Sort Key: t.a
- Sort Method: quicksort Memory: NNkB
+ Sort Method: quicksort
-> Seq Scan on t (actual rows=1000 loops=1)
(10 rows)
diff --git a/src/test/regress/expected/memoize.out b/src/test/regress/expected/memoize.out
index 4ca0bd1f1e1..fbe6bed4330 100644
--- a/src/test/regress/expected/memoize.out
+++ b/src/test/regress/expected/memoize.out
@@ -21,9 +21,7 @@ begin
end if;
ln := regexp_replace(ln, 'Evictions: 0', 'Evictions: Zero');
ln := regexp_replace(ln, 'Evictions: \d+', 'Evictions: N');
- ln := regexp_replace(ln, 'Memory Usage: \d+', 'Memory Usage: N');
- ln := regexp_replace(ln, 'Heap Fetches: \d+', 'Heap Fetches: N');
- ln := regexp_replace(ln, 'loops=\d+', 'loops=N');
+ ln := regexp_replace(ln, 'loops=\d+', 'loops=N');
return next ln;
end loop;
end;
@@ -45,11 +43,10 @@ WHERE t2.unique1 < 1000;', false);
-> Memoize (actual rows=1 loops=N)
Cache Key: t2.twenty
Cache Mode: logical
- Hits: 980 Misses: 20 Evictions: Zero Overflows: 0 Memory Usage: NkB
+ Hits: 980 Misses: 20 Evictions: Zero Overflows: 0
-> Index Only Scan using tenk1_unique1 on tenk1 t1 (actual rows=1 loops=N)
Index Cond: (unique1 = t2.twenty)
- Heap Fetches: N
-(12 rows)
+(11 rows)
-- And check we get the expected results.
SELECT COUNT(*),AVG(t1.unique1) FROM tenk1 t1
@@ -75,11 +72,10 @@ WHERE t1.unique1 < 1000;', false);
-> Memoize (actual rows=1 loops=N)
Cache Key: t1.twenty
Cache Mode: logical
- Hits: 980 Misses: 20 Evictions: Zero Overflows: 0 Memory Usage: NkB
+ Hits: 980 Misses: 20 Evictions: Zero Overflows: 0
-> Index Only Scan using tenk1_unique1 on tenk1 t2 (actual rows=1 loops=N)
Index Cond: (unique1 = t1.twenty)
- Heap Fetches: N
-(12 rows)
+(11 rows)
-- And check we get the expected results.
SELECT COUNT(*),AVG(t2.unique1) FROM tenk1 t1,
@@ -110,11 +106,10 @@ WHERE t2.unique1 < 1200;', true);
-> Memoize (actual rows=1 loops=N)
Cache Key: t2.thousand
Cache Mode: logical
- Hits: N Misses: N Evictions: N Overflows: 0 Memory Usage: NkB
+ Hits: N Misses: N Evictions: N Overflows: 0
-> Index Only Scan using tenk1_unique1 on tenk1 t1 (actual rows=1 loops=N)
Index Cond: (unique1 = t2.thousand)
- Heap Fetches: N
-(12 rows)
+(11 rows)
CREATE TABLE flt (f float);
CREATE INDEX flt_f_idx ON flt (f);
@@ -128,15 +123,13 @@ SELECT * FROM flt f1 INNER JOIN flt f2 ON f1.f = f2.f;', false);
-------------------------------------------------------------------------------
Nested Loop (actual rows=4 loops=N)
-> Index Only Scan using flt_f_idx on flt f1 (actual rows=2 loops=N)
- Heap Fetches: N
-> Memoize (actual rows=2 loops=N)
Cache Key: f1.f
Cache Mode: logical
- Hits: 1 Misses: 1 Evictions: Zero Overflows: 0 Memory Usage: NkB
+ Hits: 1 Misses: 1 Evictions: Zero Overflows: 0
-> Index Only Scan using flt_f_idx on flt f2 (actual rows=2 loops=N)
Index Cond: (f = f1.f)
- Heap Fetches: N
-(10 rows)
+(8 rows)
-- Ensure memoize operates in binary mode
SELECT explain_memoize('
@@ -145,15 +138,13 @@ SELECT * FROM flt f1 INNER JOIN flt f2 ON f1.f >= f2.f;', false);
-------------------------------------------------------------------------------
Nested Loop (actual rows=4 loops=N)
-> Index Only Scan using flt_f_idx on flt f1 (actual rows=2 loops=N)
- Heap Fetches: N
-> Memoize (actual rows=2 loops=N)
Cache Key: f1.f
Cache Mode: binary
- Hits: 0 Misses: 2 Evictions: Zero Overflows: 0 Memory Usage: NkB
+ Hits: 0 Misses: 2 Evictions: Zero Overflows: 0
-> Index Only Scan using flt_f_idx on flt f2 (actual rows=2 loops=N)
Index Cond: (f <= f1.f)
- Heap Fetches: N
-(10 rows)
+(8 rows)
DROP TABLE flt;
-- Exercise Memoize in binary mode with a large fixed width type and a
@@ -175,7 +166,7 @@ SELECT * FROM strtest s1 INNER JOIN strtest s2 ON s1.n >= s2.n;', false);
-> Memoize (actual rows=4 loops=N)
Cache Key: s1.n
Cache Mode: binary
- Hits: 3 Misses: 3 Evictions: Zero Overflows: 0 Memory Usage: NkB
+ Hits: 3 Misses: 3 Evictions: Zero Overflows: 0
-> Index Scan using strtest_n_idx on strtest s2 (actual rows=4 loops=N)
Index Cond: (n <= s1.n)
(8 rows)
@@ -190,7 +181,7 @@ SELECT * FROM strtest s1 INNER JOIN strtest s2 ON s1.t >= s2.t;', false);
-> Memoize (actual rows=4 loops=N)
Cache Key: s1.t
Cache Mode: binary
- Hits: 3 Misses: 3 Evictions: Zero Overflows: 0 Memory Usage: NkB
+ Hits: 3 Misses: 3 Evictions: Zero Overflows: 0
-> Index Scan using strtest_t_idx on strtest s2 (actual rows=4 loops=N)
Index Cond: (t <= s1.t)
(8 rows)
diff --git a/src/test/regress/expected/partition_prune.out b/src/test/regress/expected/partition_prune.out
index 7555764c779..cabadd48b81 100644
--- a/src/test/regress/expected/partition_prune.out
+++ b/src/test/regress/expected/partition_prune.out
@@ -2479,7 +2479,6 @@ update ab_a1 set b = 3 from ab where ab.a = 1 and ab.a = ab_a1.a;
Index Cond: (a = 1)
-> Bitmap Heap Scan on ab_a1_b2 ab_a1_2 (actual rows=1 loops=1)
Recheck Cond: (a = 1)
- Heap Blocks: exact=1
-> Bitmap Index Scan on ab_a1_b2_a_idx (actual rows=1 loops=1)
Index Cond: (a = 1)
-> Bitmap Heap Scan on ab_a1_b3 ab_a1_3 (actual rows=0 loops=1)
@@ -2494,14 +2493,13 @@ update ab_a1 set b = 3 from ab where ab.a = 1 and ab.a = ab_a1.a;
Index Cond: (a = 1)
-> Bitmap Heap Scan on ab_a1_b2 ab_2 (actual rows=1 loops=1)
Recheck Cond: (a = 1)
- Heap Blocks: exact=1
-> Bitmap Index Scan on ab_a1_b2_a_idx (actual rows=1 loops=1)
Index Cond: (a = 1)
-> Bitmap Heap Scan on ab_a1_b3 ab_3 (actual rows=0 loops=1)
Recheck Cond: (a = 1)
-> Bitmap Index Scan on ab_a1_b3_a_idx (actual rows=1 loops=1)
Index Cond: (a = 1)
-(34 rows)
+(32 rows)
table ab;
a | b
diff --git a/src/test/regress/expected/select_parallel.out b/src/test/regress/expected/select_parallel.out
index 4ea1aa7dfd4..9f36c627419 100644
--- a/src/test/regress/expected/select_parallel.out
+++ b/src/test/regress/expected/select_parallel.out
@@ -562,24 +562,11 @@ explain (analyze, timing off, summary off, costs off)
alter table tenk2 reset (parallel_workers);
reset work_mem;
-create function explain_parallel_sort_stats() returns setof text
-language plpgsql as
-$$
-declare ln text;
-begin
- for ln in
- explain (analyze, timing off, summary off, costs off)
- select * from
+explain (analyze)
+select * from
(select ten from tenk1 where ten < 100 order by ten) ss
- right join (values (1),(2),(3)) v(x) on true
- loop
- ln := regexp_replace(ln, 'Memory: \S*', 'Memory: xxx');
- return next ln;
- end loop;
-end;
-$$;
-select * from explain_parallel_sort_stats();
- explain_parallel_sort_stats
+ right join (values (1),(2),(3)) v(x) on true;
+ QUERY PLAN
--------------------------------------------------------------------------
Nested Loop Left Join (actual rows=30000 loops=1)
-> Values Scan on "*VALUES*" (actual rows=3 loops=1)
@@ -588,11 +575,11 @@ select * from explain_parallel_sort_stats();
Workers Launched: 4
-> Sort (actual rows=2000 loops=15)
Sort Key: tenk1.ten
- Sort Method: quicksort Memory: xxx
- Worker 0: Sort Method: quicksort Memory: xxx
- Worker 1: Sort Method: quicksort Memory: xxx
- Worker 2: Sort Method: quicksort Memory: xxx
- Worker 3: Sort Method: quicksort Memory: xxx
+ Sort Method: quicksort
+ Worker 0: Sort Method: quicksort
+ Worker 1: Sort Method: quicksort
+ Worker 2: Sort Method: quicksort
+ Worker 3: Sort Method: quicksort
-> Parallel Seq Scan on tenk1 (actual rows=2000 loops=15)
Filter: (ten < 100)
(14 rows)
@@ -603,7 +590,6 @@ reset enable_mergejoin;
reset enable_material;
reset effective_io_concurrency;
drop table bmscantest;
-drop function explain_parallel_sort_stats();
-- test parallel merge join path.
set enable_hashjoin to off;
set enable_nestloop to off;
diff --git a/src/test/regress/expected/subselect.out b/src/test/regress/expected/subselect.out
index 45c75eecc5f..527f04e122b 100644
--- a/src/test/regress/expected/subselect.out
+++ b/src/test/regress/expected/subselect.out
@@ -1550,27 +1550,15 @@ insert into sq_limit values
(6, 2, 2),
(7, 3, 3),
(8, 4, 4);
-create function explain_sq_limit() returns setof text language plpgsql as
-$$
-declare ln text;
-begin
- for ln in
- explain (analyze, summary off, timing off, costs off)
- select * from (select pk,c2 from sq_limit order by c1,pk) as x limit 3
- loop
- ln := regexp_replace(ln, 'Memory: \S*', 'Memory: xxx');
- return next ln;
- end loop;
-end;
-$$;
-select * from explain_sq_limit();
- explain_sq_limit
+explain (analyze)
+ select * from (select pk,c2 from sq_limit order by c1,pk) as x limit 3;
+ QUERY PLAN
----------------------------------------------------------------
Limit (actual rows=3 loops=1)
-> Subquery Scan on x (actual rows=3 loops=1)
-> Sort (actual rows=3 loops=1)
Sort Key: sq_limit.c1, sq_limit.pk
- Sort Method: top-N heapsort Memory: xxx
+ Sort Method: top-N heapsort
-> Seq Scan on sq_limit (actual rows=8 loops=1)
(6 rows)
@@ -1582,7 +1570,6 @@ select * from (select pk,c2 from sq_limit order by c1,pk) as x limit 3;
2 | 2
(3 rows)
-drop function explain_sq_limit();
drop table sq_limit;
--
-- Ensure that backward scan direction isn't propagated into
diff --git a/src/test/regress/sql/explain.sql b/src/test/regress/sql/explain.sql
index f91d1004d5e..c38857d3461 100644
--- a/src/test/regress/sql/explain.sql
+++ b/src/test/regress/sql/explain.sql
@@ -72,6 +72,23 @@ select explain_filter('explain (analyze, buffers, format yaml) select * from int
select explain_filter('explain (buffers, format text) select * from int8_tbl i8');
select explain_filter('explain (buffers, format json) select * from int8_tbl i8');
+-- HashAgg
+begin;
+SET work_mem='64kB';
+select explain_filter('explain (analyze) SELECT a, COUNT(1) FROM generate_series(1,9999)a GROUP BY 1');
+select explain_filter('explain (analyze, machine off) SELECT a, COUNT(1) FROM generate_series(1,9999)a GROUP BY 1');
+rollback;
+
+-- Bitmap scan
+begin;
+SET enable_indexscan=no;
+CREATE TABLE explainbitmap AS SELECT i AS a FROM generate_series(1,999) AS i;
+ANALYZE explainbitmap;
+CREATE INDEX ON explainbitmap(a);
+select explain_filter('explain (analyze) SELECT * FROM explainbitmap WHERE a<9');
+select explain_filter('explain (analyze, machine off) SELECT * FROM explainbitmap WHERE a<9');
+rollback;
+
-- SETTINGS option
-- We have to ignore other settings that might be imposed by the environment,
-- so printing the whole Settings field unfortunately won't do.
diff --git a/src/test/regress/sql/memoize.sql b/src/test/regress/sql/memoize.sql
index c6ed5a2aa66..1368f8dd185 100644
--- a/src/test/regress/sql/memoize.sql
+++ b/src/test/regress/sql/memoize.sql
@@ -22,9 +22,7 @@ begin
end if;
ln := regexp_replace(ln, 'Evictions: 0', 'Evictions: Zero');
ln := regexp_replace(ln, 'Evictions: \d+', 'Evictions: N');
- ln := regexp_replace(ln, 'Memory Usage: \d+', 'Memory Usage: N');
- ln := regexp_replace(ln, 'Heap Fetches: \d+', 'Heap Fetches: N');
- ln := regexp_replace(ln, 'loops=\d+', 'loops=N');
+ ln := regexp_replace(ln, 'loops=\d+', 'loops=N');
return next ln;
end loop;
end;
diff --git a/src/test/regress/sql/select_parallel.sql b/src/test/regress/sql/select_parallel.sql
index f9247312484..fc586161235 100644
--- a/src/test/regress/sql/select_parallel.sql
+++ b/src/test/regress/sql/select_parallel.sql
@@ -221,23 +221,10 @@ explain (analyze, timing off, summary off, costs off)
alter table tenk2 reset (parallel_workers);
reset work_mem;
-create function explain_parallel_sort_stats() returns setof text
-language plpgsql as
-$$
-declare ln text;
-begin
- for ln in
- explain (analyze, timing off, summary off, costs off)
- select * from
+explain (analyze)
+select * from
(select ten from tenk1 where ten < 100 order by ten) ss
- right join (values (1),(2),(3)) v(x) on true
- loop
- ln := regexp_replace(ln, 'Memory: \S*', 'Memory: xxx');
- return next ln;
- end loop;
-end;
-$$;
-select * from explain_parallel_sort_stats();
+ right join (values (1),(2),(3)) v(x) on true;
reset enable_indexscan;
reset enable_hashjoin;
@@ -245,7 +232,6 @@ reset enable_mergejoin;
reset enable_material;
reset effective_io_concurrency;
drop table bmscantest;
-drop function explain_parallel_sort_stats();
-- test parallel merge join path.
set enable_hashjoin to off;
diff --git a/src/test/regress/sql/subselect.sql b/src/test/regress/sql/subselect.sql
index 94ba91f5bb3..2311c9c0ed8 100644
--- a/src/test/regress/sql/subselect.sql
+++ b/src/test/regress/sql/subselect.sql
@@ -807,26 +807,11 @@ insert into sq_limit values
(7, 3, 3),
(8, 4, 4);
-create function explain_sq_limit() returns setof text language plpgsql as
-$$
-declare ln text;
-begin
- for ln in
- explain (analyze, summary off, timing off, costs off)
- select * from (select pk,c2 from sq_limit order by c1,pk) as x limit 3
- loop
- ln := regexp_replace(ln, 'Memory: \S*', 'Memory: xxx');
- return next ln;
- end loop;
-end;
-$$;
-
-select * from explain_sq_limit();
+explain (analyze)
+ select * from (select pk,c2 from sq_limit order by c1,pk) as x limit 3;
select * from (select pk,c2 from sq_limit order by c1,pk) as x limit 3;
-drop function explain_sq_limit();
-
drop table sq_limit;
--
--
2.17.1
[text/x-diff] 0005-f-Rows-removed-by-filter.patch (18.4K, ../../[email protected]/6-0005-f-Rows-removed-by-filter.patch)
download | inline diff:
From 6fdb83797005375b4bb7851b9f7397907c373635 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Tue, 16 Nov 2021 11:22:40 -0600
Subject: [PATCH 5/7] f!Rows removed by filter
This cleans one more kludge in partition_prune, but drags in 2 more files...
---
.../postgres_fdw/expected/postgres_fdw.out | 6 ++--
src/backend/commands/explain.c | 36 +++++++++----------
src/test/regress/expected/memoize.out | 9 ++---
src/test/regress/expected/partition_prune.out | 29 +++++----------
src/test/regress/expected/select_parallel.out | 4 +--
src/test/regress/sql/partition_prune.sql | 1 -
6 files changed, 32 insertions(+), 53 deletions(-)
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index 7d6f7d9e3df..e8e83ca5ad5 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -10496,13 +10496,12 @@ SELECT * FROM local_tbl, async_pt WHERE local_tbl.a = async_pt.a AND local_tbl.c
Nested Loop (actual rows=1 loops=1)
-> Seq Scan on local_tbl (actual rows=1 loops=1)
Filter: (c = 'bar'::text)
- Rows Removed by Filter: 1
-> Append (actual rows=1 loops=1)
-> Async Foreign Scan on async_p1 async_pt_1 (never executed)
-> Async Foreign Scan on async_p2 async_pt_2 (actual rows=1 loops=1)
-> Seq Scan on async_p3 async_pt_3 (never executed)
Filter: (local_tbl.a = a)
-(9 rows)
+(8 rows)
SELECT * FROM local_tbl, async_pt WHERE local_tbl.a = async_pt.a AND local_tbl.c = 'bar';
a | b | c | a | b | c
@@ -10636,8 +10635,7 @@ SELECT * FROM async_pt t1 WHERE t1.b === 505 LIMIT 1;
Filter: (b === 505)
-> Seq Scan on async_p3 t1_3 (actual rows=1 loops=1)
Filter: (b === 505)
- Rows Removed by Filter: 101
-(9 rows)
+(8 rows)
SELECT * FROM async_pt t1 WHERE t1.b === 505 LIMIT 1;
a | b | c
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index 8b4ff9624d4..a0212e7d241 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -1765,7 +1765,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
show_scan_qual(((IndexScan *) plan)->indexorderbyorig,
"Order By", planstate, ancestors, es);
show_scan_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
break;
@@ -1778,7 +1778,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
show_scan_qual(((IndexOnlyScan *) plan)->indexorderby,
"Order By", planstate, ancestors, es);
show_scan_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
if (es->analyze && es->machine)
@@ -1796,7 +1796,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
show_instrumentation_count("Rows Removed by Index Recheck", 2,
planstate, es);
show_scan_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
if (es->analyze)
@@ -1814,7 +1814,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
case T_WorkTableScan:
case T_SubqueryScan:
show_scan_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
break;
@@ -1823,7 +1823,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
Gather *gather = (Gather *) plan;
show_scan_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
ExplainPropertyInteger("Workers Planned", NULL,
@@ -1851,7 +1851,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
GatherMerge *gm = (GatherMerge *) plan;
show_scan_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
ExplainPropertyInteger("Workers Planned", NULL,
@@ -1889,7 +1889,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
es->verbose, es);
}
show_scan_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
break;
@@ -1903,7 +1903,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
es->verbose, es);
}
show_scan_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
break;
@@ -1919,7 +1919,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
tidquals = list_make1(make_orclause(tidquals));
show_scan_qual(tidquals, "TID Cond", planstate, ancestors, es);
show_scan_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
}
@@ -1936,14 +1936,14 @@ ExplainNode(PlanState *planstate, List *ancestors,
tidquals = list_make1(make_andclause(tidquals));
show_scan_qual(tidquals, "TID Cond", planstate, ancestors, es);
show_scan_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
}
break;
case T_ForeignScan:
show_scan_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
show_foreignscan_info((ForeignScanState *) planstate, es);
@@ -1953,7 +1953,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
CustomScanState *css = (CustomScanState *) planstate;
show_scan_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
if (css->methods->ExplainCustomScan)
@@ -1967,7 +1967,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
show_instrumentation_count("Rows Removed by Join Filter", 1,
planstate, es);
show_upper_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 2,
planstate, es);
break;
@@ -1980,7 +1980,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
show_instrumentation_count("Rows Removed by Join Filter", 1,
planstate, es);
show_upper_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 2,
planstate, es);
break;
@@ -1993,7 +1993,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
show_instrumentation_count("Rows Removed by Join Filter", 1,
planstate, es);
show_upper_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 2,
planstate, es);
break;
@@ -2001,14 +2001,14 @@ ExplainNode(PlanState *planstate, List *ancestors,
show_agg_keys(castNode(AggState, planstate), ancestors, es);
show_upper_qual(plan->qual, "Filter", planstate, ancestors, es);
show_hashagg_info((AggState *) planstate, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
break;
case T_Group:
show_group_keys(castNode(GroupState, planstate), ancestors, es);
show_upper_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
break;
@@ -2030,7 +2030,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
show_upper_qual((List *) ((Result *) plan)->resconstantqual,
"One-Time Filter", planstate, ancestors, es);
show_upper_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
break;
diff --git a/src/test/regress/expected/memoize.out b/src/test/regress/expected/memoize.out
index fbe6bed4330..3e521f3487e 100644
--- a/src/test/regress/expected/memoize.out
+++ b/src/test/regress/expected/memoize.out
@@ -39,14 +39,13 @@ WHERE t2.unique1 < 1000;', false);
-> Nested Loop (actual rows=1000 loops=N)
-> Seq Scan on tenk1 t2 (actual rows=1000 loops=N)
Filter: (unique1 < 1000)
- Rows Removed by Filter: 9000
-> Memoize (actual rows=1 loops=N)
Cache Key: t2.twenty
Cache Mode: logical
Hits: 980 Misses: 20 Evictions: Zero Overflows: 0
-> Index Only Scan using tenk1_unique1 on tenk1 t1 (actual rows=1 loops=N)
Index Cond: (unique1 = t2.twenty)
-(11 rows)
+(10 rows)
-- And check we get the expected results.
SELECT COUNT(*),AVG(t1.unique1) FROM tenk1 t1
@@ -68,14 +67,13 @@ WHERE t1.unique1 < 1000;', false);
-> Nested Loop (actual rows=1000 loops=N)
-> Seq Scan on tenk1 t1 (actual rows=1000 loops=N)
Filter: (unique1 < 1000)
- Rows Removed by Filter: 9000
-> Memoize (actual rows=1 loops=N)
Cache Key: t1.twenty
Cache Mode: logical
Hits: 980 Misses: 20 Evictions: Zero Overflows: 0
-> Index Only Scan using tenk1_unique1 on tenk1 t2 (actual rows=1 loops=N)
Index Cond: (unique1 = t1.twenty)
-(11 rows)
+(10 rows)
-- And check we get the expected results.
SELECT COUNT(*),AVG(t2.unique1) FROM tenk1 t1,
@@ -102,14 +100,13 @@ WHERE t2.unique1 < 1200;', true);
-> Nested Loop (actual rows=1200 loops=N)
-> Seq Scan on tenk1 t2 (actual rows=1200 loops=N)
Filter: (unique1 < 1200)
- Rows Removed by Filter: 8800
-> Memoize (actual rows=1 loops=N)
Cache Key: t2.thousand
Cache Mode: logical
Hits: N Misses: N Evictions: N Overflows: 0
-> Index Only Scan using tenk1_unique1 on tenk1 t1 (actual rows=1 loops=N)
Index Cond: (unique1 = t2.thousand)
-(11 rows)
+(10 rows)
CREATE TABLE flt (f float);
CREATE INDEX flt_f_idx ON flt (f);
diff --git a/src/test/regress/expected/partition_prune.out b/src/test/regress/expected/partition_prune.out
index cabadd48b81..3576e65bc29 100644
--- a/src/test/regress/expected/partition_prune.out
+++ b/src/test/regress/expected/partition_prune.out
@@ -1922,17 +1922,13 @@ explain (analyze, costs off, summary off, timing off) select * from list_part wh
Append (actual rows=0 loops=1)
-> Seq Scan on list_part1 list_part_1 (actual rows=0 loops=1)
Filter: (a = (list_part_fn(1) + a))
- Rows Removed by Filter: 1
-> Seq Scan on list_part2 list_part_2 (actual rows=0 loops=1)
Filter: (a = (list_part_fn(1) + a))
- Rows Removed by Filter: 1
-> Seq Scan on list_part3 list_part_3 (actual rows=0 loops=1)
Filter: (a = (list_part_fn(1) + a))
- Rows Removed by Filter: 1
-> Seq Scan on list_part4 list_part_4 (actual rows=0 loops=1)
Filter: (a = (list_part_fn(1) + a))
- Rows Removed by Filter: 1
-(13 rows)
+(9 rows)
rollback;
drop table list_part;
@@ -1957,7 +1953,6 @@ begin
loop
ln := regexp_replace(ln, 'Workers Launched: \d+', 'Workers Launched: N');
ln := regexp_replace(ln, 'actual rows=\d+ loops=\d+', 'actual rows=N loops=N');
- ln := regexp_replace(ln, 'Rows Removed by Filter: \d+', 'Rows Removed by Filter: N');
return next ln;
end loop;
end;
@@ -2196,7 +2191,6 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
-> Nested Loop (actual rows=N loops=N)
-> Parallel Seq Scan on lprt_a a (actual rows=N loops=N)
Filter: (a = ANY ('{1,0,0}'::integer[]))
- Rows Removed by Filter: N
-> Append (actual rows=N loops=N)
-> Index Scan using ab_a1_b1_a_idx on ab_a1_b1 ab_1 (actual rows=N loops=N)
Index Cond: (a = a.a)
@@ -2216,7 +2210,7 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
Index Cond: (a = a.a)
-> Index Scan using ab_a3_b3_a_idx on ab_a3_b3 ab_9 (never executed)
Index Cond: (a = a.a)
-(28 rows)
+(27 rows)
delete from lprt_a where a = 1;
select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on ab.a = a.a where a.a in(1, 0, 0)');
@@ -2230,7 +2224,6 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
-> Nested Loop (actual rows=N loops=N)
-> Parallel Seq Scan on lprt_a a (actual rows=N loops=N)
Filter: (a = ANY ('{1,0,0}'::integer[]))
- Rows Removed by Filter: N
-> Append (actual rows=N loops=N)
-> Index Scan using ab_a1_b1_a_idx on ab_a1_b1 ab_1 (never executed)
Index Cond: (a = a.a)
@@ -2250,7 +2243,7 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
Index Cond: (a = a.a)
-> Index Scan using ab_a3_b3_a_idx on ab_a3_b3 ab_9 (never executed)
Index Cond: (a = a.a)
-(28 rows)
+(27 rows)
reset enable_hashjoin;
reset enable_mergejoin;
@@ -2437,14 +2430,13 @@ explain (analyze, costs off, summary off, timing off) execute ab_q6(1);
Filter: ((a = $1) AND (b = $0))
-> Seq Scan on xy_1 (actual rows=0 loops=1)
Filter: ((x = $1) AND (y = $0))
- Rows Removed by Filter: 1
-> Seq Scan on ab_a1_b1 ab_4 (never executed)
Filter: ((a = $1) AND (b = $0))
-> Seq Scan on ab_a1_b2 ab_5 (never executed)
Filter: ((a = $1) AND (b = $0))
-> Seq Scan on ab_a1_b3 ab_6 (never executed)
Filter: ((a = $1) AND (b = $0))
-(19 rows)
+(18 rows)
-- Ensure we see just the xy_1 row.
execute ab_q6(100);
@@ -3052,12 +3044,11 @@ select * from boolp where a = (select value from boolvalues where value);
InitPlan 1 (returns $0)
-> Seq Scan on boolvalues (actual rows=1 loops=1)
Filter: value
- Rows Removed by Filter: 1
-> Seq Scan on boolp_f boolp_1 (never executed)
Filter: (a = $0)
-> Seq Scan on boolp_t boolp_2 (actual rows=0 loops=1)
Filter: (a = $0)
-(9 rows)
+(8 rows)
explain (analyze, costs off, summary off, timing off)
select * from boolp where a = (select value from boolvalues where not value);
@@ -3067,12 +3058,11 @@ select * from boolp where a = (select value from boolvalues where not value);
InitPlan 1 (returns $0)
-> Seq Scan on boolvalues (actual rows=1 loops=1)
Filter: (NOT value)
- Rows Removed by Filter: 1
-> Seq Scan on boolp_f boolp_1 (actual rows=0 loops=1)
Filter: (a = $0)
-> Seq Scan on boolp_t boolp_2 (never executed)
Filter: (a = $0)
-(9 rows)
+(8 rows)
drop table boolp;
--
@@ -3096,11 +3086,9 @@ explain (analyze, costs off, summary off, timing off) execute mt_q1(15);
Subplans Removed: 1
-> Index Scan using ma_test_p2_b_idx on ma_test_p2 ma_test_1 (actual rows=1 loops=1)
Filter: ((a >= $1) AND ((a % 10) = 5))
- Rows Removed by Filter: 9
-> Index Scan using ma_test_p3_b_idx on ma_test_p3 ma_test_2 (actual rows=1 loops=1)
Filter: ((a >= $1) AND ((a % 10) = 5))
- Rows Removed by Filter: 9
-(9 rows)
+(7 rows)
execute mt_q1(15);
a
@@ -3117,8 +3105,7 @@ explain (analyze, costs off, summary off, timing off) execute mt_q1(25);
Subplans Removed: 2
-> Index Scan using ma_test_p3_b_idx on ma_test_p3 ma_test_1 (actual rows=1 loops=1)
Filter: ((a >= $1) AND ((a % 10) = 5))
- Rows Removed by Filter: 9
-(6 rows)
+(5 rows)
execute mt_q1(25);
a
diff --git a/src/test/regress/expected/select_parallel.out b/src/test/regress/expected/select_parallel.out
index 9f36c627419..02176d7e2f6 100644
--- a/src/test/regress/expected/select_parallel.out
+++ b/src/test/regress/expected/select_parallel.out
@@ -551,14 +551,12 @@ explain (analyze, timing off, summary off, costs off)
-> Nested Loop (actual rows=98000 loops=1)
-> Seq Scan on tenk2 (actual rows=10 loops=1)
Filter: (thousand = 0)
- Rows Removed by Filter: 9990
-> Gather (actual rows=9800 loops=10)
Workers Planned: 4
Workers Launched: 4
-> Parallel Seq Scan on tenk1 (actual rows=1960 loops=50)
Filter: (hundred > 1)
- Rows Removed by Filter: 40
-(11 rows)
+(9 rows)
alter table tenk2 reset (parallel_workers);
reset work_mem;
diff --git a/src/test/regress/sql/partition_prune.sql b/src/test/regress/sql/partition_prune.sql
index d70bd8610cb..e3938bea9c0 100644
--- a/src/test/regress/sql/partition_prune.sql
+++ b/src/test/regress/sql/partition_prune.sql
@@ -463,7 +463,6 @@ begin
loop
ln := regexp_replace(ln, 'Workers Launched: \d+', 'Workers Launched: N');
ln := regexp_replace(ln, 'actual rows=\d+ loops=\d+', 'actual rows=N loops=N');
- ln := regexp_replace(ln, 'Rows Removed by Filter: \d+', 'Rows Removed by Filter: N');
return next ln;
end loop;
end;
--
2.17.1
[text/x-diff] 0006-f-Workers-Launched.patch (13.9K, ../../[email protected]/7-0006-f-Workers-Launched.patch)
download | inline diff:
From b1d45202a0db2e703948c3c63c6001a779c4b432 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Tue, 16 Nov 2021 10:37:45 -0600
Subject: [PATCH 6/7] f!Workers Launched: ...
---
src/backend/commands/explain.c | 4 +-
src/test/regress/expected/partition_prune.out | 38 ++++++-------------
src/test/regress/expected/select_parallel.out | 9 ++---
src/test/regress/sql/partition_prune.sql | 1 -
4 files changed, 17 insertions(+), 35 deletions(-)
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a0212e7d241..bd846170daf 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -1833,7 +1833,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
if (gather->initParam)
show_eval_params(gather->initParam, es);
- if (es->analyze)
+ if (es->analyze && es->machine)
{
int nworkers;
@@ -1861,7 +1861,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
if (gm->initParam)
show_eval_params(gm->initParam, es);
- if (es->analyze)
+ if (es->analyze && es->machine)
{
int nworkers;
diff --git a/src/test/regress/expected/partition_prune.out b/src/test/regress/expected/partition_prune.out
index 3576e65bc29..97c576decd7 100644
--- a/src/test/regress/expected/partition_prune.out
+++ b/src/test/regress/expected/partition_prune.out
@@ -1951,7 +1951,6 @@ begin
execute format('explain (analyze, costs off, summary off, timing off) %s',
$1)
loop
- ln := regexp_replace(ln, 'Workers Launched: \d+', 'Workers Launched: N');
ln := regexp_replace(ln, 'actual rows=\d+ loops=\d+', 'actual rows=N loops=N');
return next ln;
end loop;
@@ -1970,7 +1969,6 @@ select explain_parallel_append('execute ab_q4 (2, 2)');
Finalize Aggregate (actual rows=N loops=N)
-> Gather (actual rows=N loops=N)
Workers Planned: 2
- Workers Launched: N
-> Partial Aggregate (actual rows=N loops=N)
-> Parallel Append (actual rows=N loops=N)
Subplans Removed: 6
@@ -1980,7 +1978,7 @@ select explain_parallel_append('execute ab_q4 (2, 2)');
Filter: ((a >= $1) AND (a <= $2) AND (b < 4))
-> Parallel Seq Scan on ab_a2_b3 ab_3 (actual rows=N loops=N)
Filter: ((a >= $1) AND (a <= $2) AND (b < 4))
-(13 rows)
+(12 rows)
-- Test run-time pruning with IN lists.
prepare ab_q5 (int, int, int) as
@@ -1991,7 +1989,6 @@ select explain_parallel_append('execute ab_q5 (1, 1, 1)');
Finalize Aggregate (actual rows=N loops=N)
-> Gather (actual rows=N loops=N)
Workers Planned: 2
- Workers Launched: N
-> Partial Aggregate (actual rows=N loops=N)
-> Parallel Append (actual rows=N loops=N)
Subplans Removed: 6
@@ -2001,7 +1998,7 @@ select explain_parallel_append('execute ab_q5 (1, 1, 1)');
Filter: ((b < 4) AND (a = ANY (ARRAY[$1, $2, $3])))
-> Parallel Seq Scan on ab_a1_b3 ab_3 (actual rows=N loops=N)
Filter: ((b < 4) AND (a = ANY (ARRAY[$1, $2, $3])))
-(13 rows)
+(12 rows)
select explain_parallel_append('execute ab_q5 (2, 3, 3)');
explain_parallel_append
@@ -2009,7 +2006,6 @@ select explain_parallel_append('execute ab_q5 (2, 3, 3)');
Finalize Aggregate (actual rows=N loops=N)
-> Gather (actual rows=N loops=N)
Workers Planned: 2
- Workers Launched: N
-> Partial Aggregate (actual rows=N loops=N)
-> Parallel Append (actual rows=N loops=N)
Subplans Removed: 3
@@ -2025,7 +2021,7 @@ select explain_parallel_append('execute ab_q5 (2, 3, 3)');
Filter: ((b < 4) AND (a = ANY (ARRAY[$1, $2, $3])))
-> Parallel Seq Scan on ab_a3_b3 ab_6 (actual rows=N loops=N)
Filter: ((b < 4) AND (a = ANY (ARRAY[$1, $2, $3])))
-(19 rows)
+(18 rows)
-- Try some params whose values do not belong to any partition.
select explain_parallel_append('execute ab_q5 (33, 44, 55)');
@@ -2034,11 +2030,10 @@ select explain_parallel_append('execute ab_q5 (33, 44, 55)');
Finalize Aggregate (actual rows=N loops=N)
-> Gather (actual rows=N loops=N)
Workers Planned: 2
- Workers Launched: N
-> Partial Aggregate (actual rows=N loops=N)
-> Parallel Append (actual rows=N loops=N)
Subplans Removed: 9
-(7 rows)
+(6 rows)
-- Test Parallel Append with PARAM_EXEC Params
select explain_parallel_append('select count(*) from ab where (a = (select 1) or a = (select 3)) and b = 2');
@@ -2052,7 +2047,6 @@ select explain_parallel_append('select count(*) from ab where (a = (select 1) or
-> Gather (actual rows=N loops=N)
Workers Planned: 2
Params Evaluated: $0, $1
- Workers Launched: N
-> Parallel Append (actual rows=N loops=N)
-> Parallel Seq Scan on ab_a1_b2 ab_1 (actual rows=N loops=N)
Filter: ((b = 2) AND ((a = $0) OR (a = $1)))
@@ -2060,7 +2054,7 @@ select explain_parallel_append('select count(*) from ab where (a = (select 1) or
Filter: ((b = 2) AND ((a = $0) OR (a = $1)))
-> Parallel Seq Scan on ab_a3_b2 ab_3 (actual rows=N loops=N)
Filter: ((b = 2) AND ((a = $0) OR (a = $1)))
-(16 rows)
+(15 rows)
-- Test pruning during parallel nested loop query
create table lprt_a (a int not null);
@@ -2087,7 +2081,6 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
Finalize Aggregate (actual rows=N loops=N)
-> Gather (actual rows=N loops=N)
Workers Planned: 1
- Workers Launched: N
-> Partial Aggregate (actual rows=N loops=N)
-> Nested Loop (actual rows=N loops=N)
-> Parallel Seq Scan on lprt_a a (actual rows=N loops=N)
@@ -2111,7 +2104,7 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
Index Cond: (a = a.a)
-> Index Scan using ab_a3_b3_a_idx on ab_a3_b3 ab_9 (never executed)
Index Cond: (a = a.a)
-(27 rows)
+(26 rows)
-- Ensure the same partitions are pruned when we make the nested loop
-- parameter an Expr rather than a plain Param.
@@ -2121,7 +2114,6 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
Finalize Aggregate (actual rows=N loops=N)
-> Gather (actual rows=N loops=N)
Workers Planned: 1
- Workers Launched: N
-> Partial Aggregate (actual rows=N loops=N)
-> Nested Loop (actual rows=N loops=N)
-> Parallel Seq Scan on lprt_a a (actual rows=N loops=N)
@@ -2145,7 +2137,7 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
Index Cond: (a = (a.a + 0))
-> Index Scan using ab_a3_b3_a_idx on ab_a3_b3 ab_9 (never executed)
Index Cond: (a = (a.a + 0))
-(27 rows)
+(26 rows)
insert into lprt_a values(3),(3);
select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on ab.a = a.a where a.a in(1, 0, 3)');
@@ -2154,7 +2146,6 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
Finalize Aggregate (actual rows=N loops=N)
-> Gather (actual rows=N loops=N)
Workers Planned: 1
- Workers Launched: N
-> Partial Aggregate (actual rows=N loops=N)
-> Nested Loop (actual rows=N loops=N)
-> Parallel Seq Scan on lprt_a a (actual rows=N loops=N)
@@ -2178,7 +2169,7 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
Index Cond: (a = a.a)
-> Index Scan using ab_a3_b3_a_idx on ab_a3_b3 ab_9 (actual rows=N loops=N)
Index Cond: (a = a.a)
-(27 rows)
+(26 rows)
select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on ab.a = a.a where a.a in(1, 0, 0)');
explain_parallel_append
@@ -2186,7 +2177,6 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
Finalize Aggregate (actual rows=N loops=N)
-> Gather (actual rows=N loops=N)
Workers Planned: 1
- Workers Launched: N
-> Partial Aggregate (actual rows=N loops=N)
-> Nested Loop (actual rows=N loops=N)
-> Parallel Seq Scan on lprt_a a (actual rows=N loops=N)
@@ -2210,7 +2200,7 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
Index Cond: (a = a.a)
-> Index Scan using ab_a3_b3_a_idx on ab_a3_b3 ab_9 (never executed)
Index Cond: (a = a.a)
-(27 rows)
+(26 rows)
delete from lprt_a where a = 1;
select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on ab.a = a.a where a.a in(1, 0, 0)');
@@ -2219,7 +2209,6 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
Finalize Aggregate (actual rows=N loops=N)
-> Gather (actual rows=N loops=N)
Workers Planned: 1
- Workers Launched: N
-> Partial Aggregate (actual rows=N loops=N)
-> Nested Loop (actual rows=N loops=N)
-> Parallel Seq Scan on lprt_a a (actual rows=N loops=N)
@@ -2243,7 +2232,7 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
Index Cond: (a = a.a)
-> Index Scan using ab_a3_b3_a_idx on ab_a3_b3 ab_9 (never executed)
Index Cond: (a = a.a)
-(27 rows)
+(26 rows)
reset enable_hashjoin;
reset enable_mergejoin;
@@ -3664,7 +3653,6 @@ select explain_parallel_append('select * from listp where a = (select 1);');
Gather (actual rows=N loops=N)
Workers Planned: 2
Params Evaluated: $0
- Workers Launched: N
InitPlan 1 (returns $0)
-> Result (actual rows=N loops=N)
-> Parallel Append (actual rows=N loops=N)
@@ -3672,7 +3660,7 @@ select explain_parallel_append('select * from listp where a = (select 1);');
Filter: (a = $0)
-> Parallel Seq Scan on listp_12_2 listp_2 (never executed)
Filter: (a = $0)
-(11 rows)
+(10 rows)
-- Like the above but throw some more complexity at the planner by adding
-- a UNION ALL. We expect both sides of the union not to scan the
@@ -3687,7 +3675,6 @@ select * from listp where a = (select 2);');
-> Gather (actual rows=N loops=N)
Workers Planned: 2
Params Evaluated: $0
- Workers Launched: N
InitPlan 1 (returns $0)
-> Result (actual rows=N loops=N)
-> Parallel Append (actual rows=N loops=N)
@@ -3698,7 +3685,6 @@ select * from listp where a = (select 2);');
-> Gather (actual rows=N loops=N)
Workers Planned: 2
Params Evaluated: $1
- Workers Launched: N
InitPlan 2 (returns $1)
-> Result (actual rows=N loops=N)
-> Parallel Append (actual rows=N loops=N)
@@ -3706,7 +3692,7 @@ select * from listp where a = (select 2);');
Filter: (a = $1)
-> Parallel Seq Scan on listp_12_2 listp_5 (actual rows=N loops=N)
Filter: (a = $1)
-(23 rows)
+(21 rows)
drop table listp;
reset parallel_tuple_cost;
diff --git a/src/test/regress/expected/select_parallel.out b/src/test/regress/expected/select_parallel.out
index 02176d7e2f6..4c2e83c763e 100644
--- a/src/test/regress/expected/select_parallel.out
+++ b/src/test/regress/expected/select_parallel.out
@@ -553,10 +553,9 @@ explain (analyze, timing off, summary off, costs off)
Filter: (thousand = 0)
-> Gather (actual rows=9800 loops=10)
Workers Planned: 4
- Workers Launched: 4
-> Parallel Seq Scan on tenk1 (actual rows=1960 loops=50)
Filter: (hundred > 1)
-(9 rows)
+(8 rows)
alter table tenk2 reset (parallel_workers);
reset work_mem;
@@ -570,7 +569,6 @@ select * from
-> Values Scan on "*VALUES*" (actual rows=3 loops=1)
-> Gather Merge (actual rows=10000 loops=3)
Workers Planned: 4
- Workers Launched: 4
-> Sort (actual rows=2000 loops=15)
Sort Key: tenk1.ten
Sort Method: quicksort
@@ -580,7 +578,7 @@ select * from
Worker 3: Sort Method: quicksort
-> Parallel Seq Scan on tenk1 (actual rows=2000 loops=15)
Filter: (ten < 100)
-(14 rows)
+(13 rows)
reset enable_indexscan;
reset enable_hashjoin;
@@ -1032,9 +1030,8 @@ EXPLAIN (analyze, timing off, summary off, costs off) SELECT * FROM tenk1;
-------------------------------------------------------------
Gather (actual rows=10000 loops=1)
Workers Planned: 4
- Workers Launched: 4
-> Parallel Seq Scan on tenk1 (actual rows=2000 loops=5)
-(4 rows)
+(3 rows)
ROLLBACK TO SAVEPOINT settings;
-- provoke error in worker
diff --git a/src/test/regress/sql/partition_prune.sql b/src/test/regress/sql/partition_prune.sql
index e3938bea9c0..24908a91f6c 100644
--- a/src/test/regress/sql/partition_prune.sql
+++ b/src/test/regress/sql/partition_prune.sql
@@ -461,7 +461,6 @@ begin
execute format('explain (analyze, costs off, summary off, timing off) %s',
$1)
loop
- ln := regexp_replace(ln, 'Workers Launched: \d+', 'Workers Launched: N');
ln := regexp_replace(ln, 'actual rows=\d+ loops=\d+', 'actual rows=N loops=N');
return next ln;
end loop;
--
2.17.1
[text/x-diff] 0007-f-parallel-rows.patch (2.0K, ../../[email protected]/8-0007-f-parallel-rows.patch)
download | inline diff:
From da5503935475f80e1ce6498fb3fc283e341f67d5 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Tue, 16 Nov 2021 10:51:39 -0600
Subject: [PATCH 7/7] f!parallel rows
---
src/backend/commands/explain.c | 13 +++++++++----
1 file changed, 9 insertions(+), 4 deletions(-)
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index bd846170daf..b824fadddcd 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -1645,6 +1645,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
" (actual time=%.3f..%.3f rows=%.0f loops=%.0f)",
startup_ms, total_ms, rows, nloops);
else
+ /* This is always shown for nonparallel output */
appendStringInfo(es->str,
" (actual rows=%.0f loops=%.0f)",
rows, nloops);
@@ -1673,8 +1674,12 @@ ExplainNode(PlanState *planstate, List *ancestors,
ExplainPropertyFloat("Actual Startup Time", "ms", 0.0, 3, es);
ExplainPropertyFloat("Actual Total Time", "ms", 0.0, 3, es);
}
- ExplainPropertyFloat("Actual Rows", NULL, 0.0, 0, es);
- ExplainPropertyFloat("Actual Loops", NULL, 0.0, 0, es);
+
+ if (es->machine)
+ {
+ ExplainPropertyFloat("Actual Rows", NULL, 0.0, 0, es);
+ ExplainPropertyFloat("Actual Loops", NULL, 0.0, 0, es);
+ }
}
}
@@ -1710,7 +1715,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
appendStringInfo(es->str,
"actual time=%.3f..%.3f rows=%.0f loops=%.0f\n",
startup_ms, total_ms, rows, nloops);
- else
+ else if (es->machine)
appendStringInfo(es->str,
"actual rows=%.0f loops=%.0f\n",
rows, nloops);
@@ -1724,7 +1729,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
ExplainPropertyFloat("Actual Total Time", "ms",
total_ms, 3, es);
}
- ExplainPropertyFloat("Actual Rows", NULL, rows, 0, es);
+ ExplainPropertyFloat("Actual Rows", NULL, rows, 0, es); //
ExplainPropertyFloat("Actual Loops", NULL, nloops, 0, es);
}
--
2.17.1
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE))
@ 2022-02-26 21:07 Justin Pryzby <[email protected]>
parent: Justin Pryzby <[email protected]>
1 sibling, 1 reply; 14+ messages in thread
From: Justin Pryzby @ 2022-02-26 21:07 UTC (permalink / raw)
To: Nikolay Samokhvalov <[email protected]>; +Cc: [email protected]
Rebased over ebf6c5249b7db525e59563fb149642665c88f747.
It looks like that patch handles only query_id, and this patch also tries to
handle a bunch of other stuff.
If it's helpful, feel free to kick this patch to a future CF.
Attachments:
[text/x-diff] 0001-Add-GUC-explain_regress.patch (9.8K, ../../[email protected]/2-0001-Add-GUC-explain_regress.patch)
download | inline diff:
From e58fffedc6f1cf471228fb3234faba35898678c3 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Sat, 22 Feb 2020 21:17:10 -0600
Subject: [PATCH 1/6] Add GUC: explain_regress
This changes the defaults for explain to: costs off, timing off, summary off.
It'd be reasonable to use this for new regression tests which are not intended
to be backpatched.
---
contrib/postgres_fdw/postgres_fdw.c | 2 +-
src/backend/commands/explain.c | 13 +++++++++++--
src/backend/utils/misc/guc.c | 13 +++++++++++++
src/bin/psql/describe.c | 2 +-
src/include/commands/explain.h | 2 ++
src/test/regress/expected/explain.out | 3 +++
src/test/regress/expected/inherit.out | 2 +-
src/test/regress/expected/stats_ext.out | 2 +-
src/test/regress/pg_regress.c | 3 ++-
src/test/regress/sql/explain.sql | 4 ++++
src/test/regress/sql/inherit.sql | 2 +-
src/test/regress/sql/stats_ext.sql | 2 +-
12 files changed, 41 insertions(+), 9 deletions(-)
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index 56654844e8..54da40d662 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -3124,7 +3124,7 @@ estimate_path_cost_size(PlannerInfo *root,
* values, so don't request params_list.
*/
initStringInfo(&sql);
- appendStringInfoString(&sql, "EXPLAIN ");
+ appendStringInfoString(&sql, "EXPLAIN (COSTS)");
deparseSelectStmtForRel(&sql, root, foreignrel, fdw_scan_tlist,
remote_conds, pathkeys,
fpextra ? fpextra->has_final_sort : false,
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index de81379da3..22a3cf6349 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -154,6 +154,7 @@ static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
+bool explain_regress = false; /* GUC */
/*
@@ -172,6 +173,7 @@ ExplainQuery(ParseState *pstate, ExplainStmt *stmt,
ListCell *lc;
bool timing_set = false;
bool summary_set = false;
+ bool costs_set = false;
/* Parse options list. */
foreach(lc, stmt->options)
@@ -183,7 +185,11 @@ ExplainQuery(ParseState *pstate, ExplainStmt *stmt,
else if (strcmp(opt->defname, "verbose") == 0)
es->verbose = defGetBoolean(opt);
else if (strcmp(opt->defname, "costs") == 0)
+ {
+ /* Need to keep track if it was explicitly set to ON */
+ costs_set = true;
es->costs = defGetBoolean(opt);
+ }
else if (strcmp(opt->defname, "buffers") == 0)
es->buffers = defGetBoolean(opt);
else if (strcmp(opt->defname, "wal") == 0)
@@ -227,13 +233,16 @@ ExplainQuery(ParseState *pstate, ExplainStmt *stmt,
parser_errposition(pstate, opt->location)));
}
+ /* if the costs option was not set explicitly, set default value */
+ es->costs = (costs_set) ? es->costs : es->costs && !explain_regress;
+
if (es->wal && !es->analyze)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("EXPLAIN option WAL requires ANALYZE")));
/* if the timing was not set explicitly, set default value */
- es->timing = (timing_set) ? es->timing : es->analyze;
+ es->timing = (timing_set) ? es->timing : es->analyze && !explain_regress;
/* check that timing is used with EXPLAIN ANALYZE */
if (es->timing && !es->analyze)
@@ -242,7 +251,7 @@ ExplainQuery(ParseState *pstate, ExplainStmt *stmt,
errmsg("EXPLAIN option TIMING requires ANALYZE")));
/* if the summary was not set explicitly, set default value */
- es->summary = (summary_set) ? es->summary : es->analyze;
+ es->summary = (summary_set) ? es->summary : es->analyze && !explain_regress;
query = castNode(Query, stmt->query);
if (IsQueryIdEnabled())
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 1e3650184b..87266cf7bd 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -46,6 +46,7 @@
#include "catalog/pg_authid.h"
#include "catalog/storage.h"
#include "commands/async.h"
+#include "commands/explain.h"
#include "commands/prepare.h"
#include "commands/tablespace.h"
#include "commands/trigger.h"
@@ -1474,6 +1475,18 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+
+ {
+ {"explain_regress", PGC_USERSET, DEVELOPER_OPTIONS,
+ gettext_noop("Change defaults of EXPLAIN to avoid unstable output."),
+ NULL,
+ GUC_NOT_IN_SAMPLE | GUC_EXPLAIN
+ },
+ &explain_regress,
+ false,
+ NULL, NULL, NULL
+ },
+
{
{"log_parser_stats", PGC_SUSET, STATS_MONITORING,
gettext_noop("Writes parser performance statistics to the server log."),
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index e3382933d9..8f86fbbb00 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2628,7 +2628,7 @@ describeOneTableDetails(const char *schemaname,
"stxstattarget\n"
"FROM pg_catalog.pg_statistic_ext\n"
"WHERE stxrelid = '%s'\n"
- "ORDER BY nsp, stxname;",
+ "ORDER BY nsp, stxname",
oid);
result = PSQLexec(buf.data);
diff --git a/src/include/commands/explain.h b/src/include/commands/explain.h
index 666977fb1f..ce8c458d73 100644
--- a/src/include/commands/explain.h
+++ b/src/include/commands/explain.h
@@ -61,6 +61,8 @@ typedef struct ExplainState
ExplainWorkersState *workers_state; /* needed if parallel plan */
} ExplainState;
+extern bool explain_regress;
+
/* Hook for plugins to get control in ExplainOneQuery() */
typedef void (*ExplainOneQuery_hook_type) (Query *query,
int cursorOptions,
diff --git a/src/test/regress/expected/explain.out b/src/test/regress/expected/explain.out
index 1734dfee8c..188dc7ccec 100644
--- a/src/test/regress/expected/explain.out
+++ b/src/test/regress/expected/explain.out
@@ -8,6 +8,9 @@
-- To produce stable regression test output, it's usually necessary to
-- ignore details such as exact costs or row counts. These filter
-- functions replace changeable output details with fixed strings.
+-- Output normal, user-facing details, not the sanitized version used for the
+-- rest of the regression tests
+set explain_regress = off;
create function explain_filter(text) returns setof text
language plpgsql as
$$
diff --git a/src/test/regress/expected/inherit.out b/src/test/regress/expected/inherit.out
index 2d49e765de..38fb5f94c6 100644
--- a/src/test/regress/expected/inherit.out
+++ b/src/test/regress/expected/inherit.out
@@ -664,7 +664,7 @@ select tableoid::regclass::text as relname, parted_tab.* from parted_tab order b
(3 rows)
-- modifies partition key, but no rows will actually be updated
-explain update parted_tab set a = 2 where false;
+explain (costs on) update parted_tab set a = 2 where false;
QUERY PLAN
--------------------------------------------------------
Update on parted_tab (cost=0.00..0.00 rows=0 width=0)
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 042316aeed..638abaaf80 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -14,7 +14,7 @@ declare
first_row bool := true;
begin
for ln in
- execute format('explain analyze %s', $1)
+ execute format('explain (analyze, costs on) %s', $1)
loop
if first_row then
first_row := false;
diff --git a/src/test/regress/pg_regress.c b/src/test/regress/pg_regress.c
index db8427dd9b..01f820752d 100644
--- a/src/test/regress/pg_regress.c
+++ b/src/test/regress/pg_regress.c
@@ -634,7 +634,7 @@ initialize_environment(void)
* user's ability to set other variables through that.
*/
{
- const char *my_pgoptions = "-c intervalstyle=postgres_verbose";
+ const char *my_pgoptions = "-c intervalstyle=postgres_verbose -c explain_regress=on";
const char *old_pgoptions = getenv("PGOPTIONS");
char *new_pgoptions;
@@ -2296,6 +2296,7 @@ regression_main(int argc, char *argv[],
fputs("log_lock_waits = on\n", pg_conf);
fputs("log_temp_files = 128kB\n", pg_conf);
fputs("max_prepared_transactions = 2\n", pg_conf);
+ // fputs("explain_regress = yes\n", pg_conf);
for (sl = temp_configs; sl != NULL; sl = sl->next)
{
diff --git a/src/test/regress/sql/explain.sql b/src/test/regress/sql/explain.sql
index 84549c78fa..f91d1004d5 100644
--- a/src/test/regress/sql/explain.sql
+++ b/src/test/regress/sql/explain.sql
@@ -10,6 +10,10 @@
-- ignore details such as exact costs or row counts. These filter
-- functions replace changeable output details with fixed strings.
+-- Output normal, user-facing details, not the sanitized version used for the
+-- rest of the regression tests
+set explain_regress = off;
+
create function explain_filter(text) returns setof text
language plpgsql as
$$
diff --git a/src/test/regress/sql/inherit.sql b/src/test/regress/sql/inherit.sql
index 195aedb5ff..868ee58b80 100644
--- a/src/test/regress/sql/inherit.sql
+++ b/src/test/regress/sql/inherit.sql
@@ -169,7 +169,7 @@ where parted_tab.a = ss.a;
select tableoid::regclass::text as relname, parted_tab.* from parted_tab order by 1,2;
-- modifies partition key, but no rows will actually be updated
-explain update parted_tab set a = 2 where false;
+explain (costs on) update parted_tab set a = 2 where false;
drop table parted_tab;
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 6b954c9e50..4c91f60863 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -16,7 +16,7 @@ declare
first_row bool := true;
begin
for ln in
- execute format('explain analyze %s', $1)
+ execute format('explain (analyze, costs on) %s', $1)
loop
if first_row then
first_row := false;
--
2.17.1
[text/x-diff] 0002-exercise-explain_regress.patch (12.8K, ../../[email protected]/3-0002-exercise-explain_regress.patch)
download | inline diff:
From f75e72b2489a0c7ce967972c0c181f8b8b5a61d3 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Mon, 15 Nov 2021 21:54:12 -0600
Subject: [PATCH 2/6] exercise explain_regress
not intended to be merged, since it creates backpatch hazards (unless the GUC
is also backpatched)
---
src/test/regress/expected/matview.out | 12 ++++++------
src/test/regress/expected/select_into.out | 20 ++++++++++----------
src/test/regress/expected/tidscan.out | 6 +++---
src/test/regress/sql/matview.sql | 12 ++++++------
src/test/regress/sql/select_into.sql | 20 ++++++++++----------
src/test/regress/sql/tidscan.sql | 6 +++---
6 files changed, 38 insertions(+), 38 deletions(-)
diff --git a/src/test/regress/expected/matview.out b/src/test/regress/expected/matview.out
index 313c72a268..e7ce5bb188 100644
--- a/src/test/regress/expected/matview.out
+++ b/src/test/regress/expected/matview.out
@@ -606,7 +606,7 @@ GRANT ALL ON SCHEMA matview_schema TO public;
SET SESSION AUTHORIZATION regress_matview_user;
CREATE MATERIALIZED VIEW matview_schema.mv_withdata1 (a) AS
SELECT generate_series(1, 10) WITH DATA;
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE MATERIALIZED VIEW matview_schema.mv_withdata2 (a) AS
SELECT generate_series(1, 10) WITH DATA;
QUERY PLAN
@@ -618,7 +618,7 @@ EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
REFRESH MATERIALIZED VIEW matview_schema.mv_withdata2;
CREATE MATERIALIZED VIEW matview_schema.mv_nodata1 (a) AS
SELECT generate_series(1, 10) WITH NO DATA;
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE MATERIALIZED VIEW matview_schema.mv_nodata2 (a) AS
SELECT generate_series(1, 10) WITH NO DATA;
QUERY PLAN
@@ -651,11 +651,11 @@ ERROR: relation "matview_ine_tab" already exists
CREATE MATERIALIZED VIEW IF NOT EXISTS matview_ine_tab AS
SELECT 1 / 0 WITH NO DATA; -- ok
NOTICE: relation "matview_ine_tab" already exists, skipping
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE MATERIALIZED VIEW matview_ine_tab AS
SELECT 1 / 0; -- error
ERROR: relation "matview_ine_tab" already exists
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE MATERIALIZED VIEW IF NOT EXISTS matview_ine_tab AS
SELECT 1 / 0; -- ok
NOTICE: relation "matview_ine_tab" already exists, skipping
@@ -663,11 +663,11 @@ NOTICE: relation "matview_ine_tab" already exists, skipping
------------
(0 rows)
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE MATERIALIZED VIEW matview_ine_tab AS
SELECT 1 / 0 WITH NO DATA; -- error
ERROR: relation "matview_ine_tab" already exists
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE MATERIALIZED VIEW IF NOT EXISTS matview_ine_tab AS
SELECT 1 / 0 WITH NO DATA; -- ok
NOTICE: relation "matview_ine_tab" already exists, skipping
diff --git a/src/test/regress/expected/select_into.out b/src/test/regress/expected/select_into.out
index b79fe9a1c0..03f2e9e158 100644
--- a/src/test/regress/expected/select_into.out
+++ b/src/test/regress/expected/select_into.out
@@ -25,7 +25,7 @@ CREATE TABLE selinto_schema.tbl_withdata1 (a)
AS SELECT generate_series(1,3) WITH DATA;
INSERT INTO selinto_schema.tbl_withdata1 VALUES (4);
ERROR: permission denied for table tbl_withdata1
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE selinto_schema.tbl_withdata2 (a) AS
SELECT generate_series(1,3) WITH DATA;
QUERY PLAN
@@ -37,7 +37,7 @@ EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
-- WITH NO DATA, passes.
CREATE TABLE selinto_schema.tbl_nodata1 (a) AS
SELECT generate_series(1,3) WITH NO DATA;
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE selinto_schema.tbl_nodata2 (a) AS
SELECT generate_series(1,3) WITH NO DATA;
QUERY PLAN
@@ -50,7 +50,7 @@ EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
PREPARE data_sel AS SELECT generate_series(1,3);
CREATE TABLE selinto_schema.tbl_withdata3 (a) AS
EXECUTE data_sel WITH DATA;
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE selinto_schema.tbl_withdata4 (a) AS
EXECUTE data_sel WITH DATA;
QUERY PLAN
@@ -62,7 +62,7 @@ EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
-- EXECUTE and WITH NO DATA, passes.
CREATE TABLE selinto_schema.tbl_nodata3 (a) AS
EXECUTE data_sel WITH NO DATA;
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE selinto_schema.tbl_nodata4 (a) AS
EXECUTE data_sel WITH NO DATA;
QUERY PLAN
@@ -188,20 +188,20 @@ CREATE TABLE ctas_ine_tbl AS SELECT 1 / 0 WITH NO DATA; -- error
ERROR: relation "ctas_ine_tbl" already exists
CREATE TABLE IF NOT EXISTS ctas_ine_tbl AS SELECT 1 / 0 WITH NO DATA; -- ok
NOTICE: relation "ctas_ine_tbl" already exists, skipping
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE ctas_ine_tbl AS SELECT 1 / 0; -- error
ERROR: relation "ctas_ine_tbl" already exists
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE IF NOT EXISTS ctas_ine_tbl AS SELECT 1 / 0; -- ok
NOTICE: relation "ctas_ine_tbl" already exists, skipping
QUERY PLAN
------------
(0 rows)
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE ctas_ine_tbl AS SELECT 1 / 0 WITH NO DATA; -- error
ERROR: relation "ctas_ine_tbl" already exists
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE IF NOT EXISTS ctas_ine_tbl AS SELECT 1 / 0 WITH NO DATA; -- ok
NOTICE: relation "ctas_ine_tbl" already exists, skipping
QUERY PLAN
@@ -209,10 +209,10 @@ NOTICE: relation "ctas_ine_tbl" already exists, skipping
(0 rows)
PREPARE ctas_ine_query AS SELECT 1 / 0;
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE ctas_ine_tbl AS EXECUTE ctas_ine_query; -- error
ERROR: relation "ctas_ine_tbl" already exists
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE IF NOT EXISTS ctas_ine_tbl AS EXECUTE ctas_ine_query; -- ok
NOTICE: relation "ctas_ine_tbl" already exists, skipping
QUERY PLAN
diff --git a/src/test/regress/expected/tidscan.out b/src/test/regress/expected/tidscan.out
index 13c3c360c2..de93145bf0 100644
--- a/src/test/regress/expected/tidscan.out
+++ b/src/test/regress/expected/tidscan.out
@@ -189,7 +189,7 @@ FETCH NEXT FROM c;
(1 row)
-- perform update
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
UPDATE tidscan SET id = -id WHERE CURRENT OF c RETURNING *;
QUERY PLAN
---------------------------------------------------
@@ -205,7 +205,7 @@ FETCH NEXT FROM c;
(1 row)
-- perform update
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
UPDATE tidscan SET id = -id WHERE CURRENT OF c RETURNING *;
QUERY PLAN
---------------------------------------------------
@@ -229,7 +229,7 @@ FETCH NEXT FROM c;
(0 rows)
-- should error out
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
UPDATE tidscan SET id = -id WHERE CURRENT OF c RETURNING *;
ERROR: cursor "c" is not positioned on a row
ROLLBACK;
diff --git a/src/test/regress/sql/matview.sql b/src/test/regress/sql/matview.sql
index 68b9ccfd45..e0b562933d 100644
--- a/src/test/regress/sql/matview.sql
+++ b/src/test/regress/sql/matview.sql
@@ -255,13 +255,13 @@ GRANT ALL ON SCHEMA matview_schema TO public;
SET SESSION AUTHORIZATION regress_matview_user;
CREATE MATERIALIZED VIEW matview_schema.mv_withdata1 (a) AS
SELECT generate_series(1, 10) WITH DATA;
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE MATERIALIZED VIEW matview_schema.mv_withdata2 (a) AS
SELECT generate_series(1, 10) WITH DATA;
REFRESH MATERIALIZED VIEW matview_schema.mv_withdata2;
CREATE MATERIALIZED VIEW matview_schema.mv_nodata1 (a) AS
SELECT generate_series(1, 10) WITH NO DATA;
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE MATERIALIZED VIEW matview_schema.mv_nodata2 (a) AS
SELECT generate_series(1, 10) WITH NO DATA;
REFRESH MATERIALIZED VIEW matview_schema.mv_nodata2;
@@ -282,16 +282,16 @@ CREATE MATERIALIZED VIEW matview_ine_tab AS
SELECT 1 / 0 WITH NO DATA; -- error
CREATE MATERIALIZED VIEW IF NOT EXISTS matview_ine_tab AS
SELECT 1 / 0 WITH NO DATA; -- ok
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE MATERIALIZED VIEW matview_ine_tab AS
SELECT 1 / 0; -- error
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE MATERIALIZED VIEW IF NOT EXISTS matview_ine_tab AS
SELECT 1 / 0; -- ok
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE MATERIALIZED VIEW matview_ine_tab AS
SELECT 1 / 0 WITH NO DATA; -- error
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE MATERIALIZED VIEW IF NOT EXISTS matview_ine_tab AS
SELECT 1 / 0 WITH NO DATA; -- ok
DROP MATERIALIZED VIEW matview_ine_tab;
diff --git a/src/test/regress/sql/select_into.sql b/src/test/regress/sql/select_into.sql
index 689c448cc2..85bfb2bf16 100644
--- a/src/test/regress/sql/select_into.sql
+++ b/src/test/regress/sql/select_into.sql
@@ -30,26 +30,26 @@ SET SESSION AUTHORIZATION regress_selinto_user;
CREATE TABLE selinto_schema.tbl_withdata1 (a)
AS SELECT generate_series(1,3) WITH DATA;
INSERT INTO selinto_schema.tbl_withdata1 VALUES (4);
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE selinto_schema.tbl_withdata2 (a) AS
SELECT generate_series(1,3) WITH DATA;
-- WITH NO DATA, passes.
CREATE TABLE selinto_schema.tbl_nodata1 (a) AS
SELECT generate_series(1,3) WITH NO DATA;
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE selinto_schema.tbl_nodata2 (a) AS
SELECT generate_series(1,3) WITH NO DATA;
-- EXECUTE and WITH DATA, passes.
PREPARE data_sel AS SELECT generate_series(1,3);
CREATE TABLE selinto_schema.tbl_withdata3 (a) AS
EXECUTE data_sel WITH DATA;
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE selinto_schema.tbl_withdata4 (a) AS
EXECUTE data_sel WITH DATA;
-- EXECUTE and WITH NO DATA, passes.
CREATE TABLE selinto_schema.tbl_nodata3 (a) AS
EXECUTE data_sel WITH NO DATA;
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE selinto_schema.tbl_nodata4 (a) AS
EXECUTE data_sel WITH NO DATA;
RESET SESSION AUTHORIZATION;
@@ -122,17 +122,17 @@ CREATE TABLE ctas_ine_tbl AS SELECT 1 / 0; -- error
CREATE TABLE IF NOT EXISTS ctas_ine_tbl AS SELECT 1 / 0; -- ok
CREATE TABLE ctas_ine_tbl AS SELECT 1 / 0 WITH NO DATA; -- error
CREATE TABLE IF NOT EXISTS ctas_ine_tbl AS SELECT 1 / 0 WITH NO DATA; -- ok
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE ctas_ine_tbl AS SELECT 1 / 0; -- error
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE IF NOT EXISTS ctas_ine_tbl AS SELECT 1 / 0; -- ok
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE ctas_ine_tbl AS SELECT 1 / 0 WITH NO DATA; -- error
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE IF NOT EXISTS ctas_ine_tbl AS SELECT 1 / 0 WITH NO DATA; -- ok
PREPARE ctas_ine_query AS SELECT 1 / 0;
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE ctas_ine_tbl AS EXECUTE ctas_ine_query; -- error
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
CREATE TABLE IF NOT EXISTS ctas_ine_tbl AS EXECUTE ctas_ine_query; -- ok
DROP TABLE ctas_ine_tbl;
diff --git a/src/test/regress/sql/tidscan.sql b/src/test/regress/sql/tidscan.sql
index 313e0fb9b6..3d1f447088 100644
--- a/src/test/regress/sql/tidscan.sql
+++ b/src/test/regress/sql/tidscan.sql
@@ -68,17 +68,17 @@ DECLARE c CURSOR FOR SELECT ctid, * FROM tidscan;
FETCH NEXT FROM c; -- skip one row
FETCH NEXT FROM c;
-- perform update
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
UPDATE tidscan SET id = -id WHERE CURRENT OF c RETURNING *;
FETCH NEXT FROM c;
-- perform update
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
UPDATE tidscan SET id = -id WHERE CURRENT OF c RETURNING *;
SELECT * FROM tidscan;
-- position cursor past any rows
FETCH NEXT FROM c;
-- should error out
-EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
+EXPLAIN (ANALYZE)
UPDATE tidscan SET id = -id WHERE CURRENT OF c RETURNING *;
ROLLBACK;
--
2.17.1
[text/x-diff] 0003-Make-explain-default-to-BUFFERS-TRUE.patch (4.7K, ../../[email protected]/4-0003-Make-explain-default-to-BUFFERS-TRUE.patch)
download | inline diff:
From aeddb8da4957d1d9469d46dd3372a699ef1e5350 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Wed, 22 Jul 2020 19:20:40 -0500
Subject: [PATCH 3/6] Make explain default to BUFFERS TRUE
---
contrib/auto_explain/auto_explain.c | 4 ++--
doc/src/sgml/config.sgml | 2 +-
doc/src/sgml/perform.sgml | 4 ++--
doc/src/sgml/ref/explain.sgml | 2 +-
src/backend/commands/explain.c | 8 ++++++++
5 files changed, 14 insertions(+), 6 deletions(-)
diff --git a/contrib/auto_explain/auto_explain.c b/contrib/auto_explain/auto_explain.c
index d3029f85ef..236e12596c 100644
--- a/contrib/auto_explain/auto_explain.c
+++ b/contrib/auto_explain/auto_explain.c
@@ -27,7 +27,7 @@ PG_MODULE_MAGIC;
static int auto_explain_log_min_duration = -1; /* msec or -1 */
static bool auto_explain_log_analyze = false;
static bool auto_explain_log_verbose = false;
-static bool auto_explain_log_buffers = false;
+static bool auto_explain_log_buffers = true;
static bool auto_explain_log_wal = false;
static bool auto_explain_log_triggers = false;
static bool auto_explain_log_timing = true;
@@ -143,7 +143,7 @@ _PG_init(void)
"Log buffers usage.",
NULL,
&auto_explain_log_buffers,
- false,
+ true,
PGC_SUSET,
0,
NULL,
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 7ed8c82a9d..b3a6f55ba4 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -7833,7 +7833,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
displayed in <link linkend="monitoring-pg-stat-database-view">
<structname>pg_stat_database</structname></link>, in the output of
<xref linkend="sql-explain"/> when the <literal>BUFFERS</literal> option
- is used, in the output of <xref linkend="sql-vacuum"/> when
+ is enabled, in the output of <xref linkend="sql-vacuum"/> when
the <literal>VERBOSE</literal> option is used, by autovacuum
for auto-vacuums and auto-analyzes, when <xref
linkend="guc-log-autovacuum-min-duration"/> is set and by
diff --git a/doc/src/sgml/perform.sgml b/doc/src/sgml/perform.sgml
index 9cf8ebea80..6e815e3a9a 100644
--- a/doc/src/sgml/perform.sgml
+++ b/doc/src/sgml/perform.sgml
@@ -731,8 +731,8 @@ EXPLAIN ANALYZE SELECT * FROM polygon_tbl WHERE f1 @> polygon '(0.5,2.0)';
</para>
<para>
- <command>EXPLAIN</command> has a <literal>BUFFERS</literal> option that can be used with
- <literal>ANALYZE</literal> to get even more run time statistics:
+ <command>EXPLAIN ANALYZE</command> has a <literal>BUFFERS</literal> option which
+ provides even more run time statistics:
<screen>
EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM tenk1 WHERE unique1 < 100 AND unique2 > 9000;
diff --git a/doc/src/sgml/ref/explain.sgml b/doc/src/sgml/ref/explain.sgml
index 4d758fb237..ceb0f4c83a 100644
--- a/doc/src/sgml/ref/explain.sgml
+++ b/doc/src/sgml/ref/explain.sgml
@@ -190,7 +190,7 @@ ROLLBACK;
The number of blocks shown for an
upper-level node includes those used by all its child nodes. In text
format, only non-zero values are printed. It defaults to
- <literal>FALSE</literal>.
+ <literal>TRUE</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index 22a3cf6349..d953ddf5be 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -174,6 +174,7 @@ ExplainQuery(ParseState *pstate, ExplainStmt *stmt,
bool timing_set = false;
bool summary_set = false;
bool costs_set = false;
+ bool buffers_set = false;
/* Parse options list. */
foreach(lc, stmt->options)
@@ -191,7 +192,10 @@ ExplainQuery(ParseState *pstate, ExplainStmt *stmt,
es->costs = defGetBoolean(opt);
}
else if (strcmp(opt->defname, "buffers") == 0)
+ {
+ buffers_set = true;
es->buffers = defGetBoolean(opt);
+ }
else if (strcmp(opt->defname, "wal") == 0)
es->wal = defGetBoolean(opt);
else if (strcmp(opt->defname, "settings") == 0)
@@ -253,6 +257,9 @@ ExplainQuery(ParseState *pstate, ExplainStmt *stmt,
/* if the summary was not set explicitly, set default value */
es->summary = (summary_set) ? es->summary : es->analyze && !explain_regress;
+ /* if the buffers option was not set explicitly, set default value */
+ es->buffers = (buffers_set) ? es->buffers : !explain_regress;
+
query = castNode(Query, stmt->query);
if (IsQueryIdEnabled())
jstate = JumbleQuery(query, pstate->p_sourcetext);
@@ -323,6 +330,7 @@ NewExplainState(void)
/* Set default options (most fields can be left as zeroes). */
es->costs = true;
+ es->buffers = true;
/* Prepare output buffer. */
es->str = makeStringInfo();
--
2.17.1
[text/x-diff] 0004-Add-explain-MACHINE-to-hide-machine-dependent-output.patch (33.9K, ../../[email protected]/5-0004-Add-explain-MACHINE-to-hide-machine-dependent-output.patch)
download | inline diff:
From 3577649a7b3458bb7dd790916966f186134bf29b Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Sat, 22 Feb 2020 18:45:22 -0600
Subject: [PATCH 4/6] Add explain(MACHINE) to hide machine-dependent output..
This new option hides some output that has traditionally been shown; the option
is enabled by regression mode to hide unstable output.
This allows EXPLAIN ANALYZE to be used in regression tests with stable output.
This is like a "quiet" mode, or negative verbosity.
Also add regression tests for HashAgg and Bitmap scan, which previously had no
tests with explain(analyze).
This does not handle variations in "Workers Launched", or other parallel worker
bits which are handled by force_parallel_mode=regress.
Also add tests for show_hashagg_info and tidbitmap, for which there's no other
test.
---
src/backend/commands/explain.c | 77 +++++++++++++------
src/include/commands/explain.h | 1 +
src/test/isolation/expected/horizons.out | 40 +++++-----
src/test/isolation/specs/horizons.spec | 2 +-
src/test/regress/expected/explain.out | 55 +++++++++++++
.../regress/expected/incremental_sort.out | 4 +-
src/test/regress/expected/memoize.out | 35 ++++-----
src/test/regress/expected/partition_prune.out | 4 +-
src/test/regress/expected/select_parallel.out | 32 +++-----
src/test/regress/expected/subselect.out | 21 +----
src/test/regress/sql/explain.sql | 17 ++++
src/test/regress/sql/memoize.sql | 4 +-
src/test/regress/sql/select_parallel.sql | 20 +----
src/test/regress/sql/subselect.sql | 19 +----
14 files changed, 182 insertions(+), 149 deletions(-)
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index d953ddf5be..71d6921a04 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -175,6 +175,7 @@ ExplainQuery(ParseState *pstate, ExplainStmt *stmt,
bool summary_set = false;
bool costs_set = false;
bool buffers_set = false;
+ bool machine_set = false;
/* Parse options list. */
foreach(lc, stmt->options)
@@ -210,6 +211,11 @@ ExplainQuery(ParseState *pstate, ExplainStmt *stmt,
summary_set = true;
es->summary = defGetBoolean(opt);
}
+ else if (strcmp(opt->defname, "machine") == 0)
+ {
+ machine_set = true;
+ es->machine = defGetBoolean(opt);
+ }
else if (strcmp(opt->defname, "format") == 0)
{
char *p = defGetString(opt);
@@ -260,6 +266,9 @@ ExplainQuery(ParseState *pstate, ExplainStmt *stmt,
/* if the buffers option was not set explicitly, set default value */
es->buffers = (buffers_set) ? es->buffers : !explain_regress;
+ /* if the machine option was not set explicitly, set default value */
+ es->machine = (machine_set) ? es->machine : !explain_regress;
+
query = castNode(Query, stmt->query);
if (IsQueryIdEnabled())
jstate = JumbleQuery(query, pstate->p_sourcetext);
@@ -627,7 +636,7 @@ ExplainOnePlan(PlannedStmt *plannedstmt, IntoClause *into, ExplainState *es,
* generated by regression test suites.
*/
if (es->verbose && plannedstmt->queryId != UINT64CONST(0) &&
- compute_query_id != COMPUTE_QUERY_ID_REGRESS)
+ compute_query_id != COMPUTE_QUERY_ID_REGRESS && es->machine)
{
/*
* Output the queryid as an int64 rather than a uint64 so we match
@@ -638,7 +647,7 @@ ExplainOnePlan(PlannedStmt *plannedstmt, IntoClause *into, ExplainState *es,
}
/* Show buffer usage in planning */
- if (bufusage)
+ if (bufusage && es->buffers)
{
ExplainOpenGroup("Planning", "Planning", true, es);
show_buffer_usage(es, bufusage, true);
@@ -1778,7 +1787,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
if (plan->qual)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
- if (es->analyze)
+ if (es->analyze && es->machine)
ExplainPropertyFloat("Heap Fetches", NULL,
planstate->instrument->ntuples2, 0, es);
break;
@@ -2752,8 +2761,12 @@ show_sort_info(SortState *sortstate, ExplainState *es)
if (es->format == EXPLAIN_FORMAT_TEXT)
{
ExplainIndentText(es);
- appendStringInfo(es->str, "Sort Method: %s %s: " INT64_FORMAT "kB\n",
- sortMethod, spaceType, spaceUsed);
+ appendStringInfo(es->str, "Sort Method: %s",
+ sortMethod);
+ if (es->machine)
+ appendStringInfo(es->str, " %s: " INT64_FORMAT "kB",
+ spaceType, spaceUsed);
+ appendStringInfoString(es->str, "\n");
}
else
{
@@ -2797,8 +2810,12 @@ show_sort_info(SortState *sortstate, ExplainState *es)
{
ExplainIndentText(es);
appendStringInfo(es->str,
- "Sort Method: %s %s: " INT64_FORMAT "kB\n",
- sortMethod, spaceType, spaceUsed);
+ "Sort Method: %s",
+ sortMethod);
+ if (es->machine)
+ appendStringInfo(es->str, " %s: " INT64_FORMAT "kB", spaceType, spaceUsed);
+
+ appendStringInfoString(es->str, "\n");
}
else
{
@@ -3086,25 +3103,26 @@ show_hash_info(HashState *hashstate, ExplainState *es)
ExplainPropertyInteger("Peak Memory Usage", "kB",
spacePeakKb, es);
}
- else if (hinstrument.nbatch_original != hinstrument.nbatch ||
- hinstrument.nbuckets_original != hinstrument.nbuckets)
+ else
{
ExplainIndentText(es);
- appendStringInfo(es->str,
- "Buckets: %d (originally %d) Batches: %d (originally %d) Memory Usage: %ldkB\n",
+ if (hinstrument.nbatch_original != hinstrument.nbatch ||
+ hinstrument.nbuckets_original != hinstrument.nbuckets)
+ appendStringInfo(es->str,
+ "Buckets: %d (originally %d) Batches: %d (originally %d)",
hinstrument.nbuckets,
hinstrument.nbuckets_original,
hinstrument.nbatch,
- hinstrument.nbatch_original,
- spacePeakKb);
- }
- else
- {
- ExplainIndentText(es);
- appendStringInfo(es->str,
- "Buckets: %d Batches: %d Memory Usage: %ldkB\n",
- hinstrument.nbuckets, hinstrument.nbatch,
- spacePeakKb);
+ hinstrument.nbatch_original);
+ else
+ appendStringInfo(es->str,
+ "Buckets: %d Batches: %d",
+ hinstrument.nbuckets, hinstrument.nbatch);
+
+ if (es->machine)
+ appendStringInfo(es->str, " Memory Usage: %ldkB", spacePeakKb);
+
+ appendStringInfoChar(es->str, '\n');
}
}
}
@@ -3188,12 +3206,16 @@ show_memoize_info(MemoizeState *mstate, List *ancestors, ExplainState *es)
{
ExplainIndentText(es);
appendStringInfo(es->str,
- "Hits: " UINT64_FORMAT " Misses: " UINT64_FORMAT " Evictions: " UINT64_FORMAT " Overflows: " UINT64_FORMAT " Memory Usage: " INT64_FORMAT "kB\n",
+ "Hits: " UINT64_FORMAT " Misses: " UINT64_FORMAT " Evictions: " UINT64_FORMAT " Overflows: " UINT64_FORMAT,
mstate->stats.cache_hits,
mstate->stats.cache_misses,
mstate->stats.cache_evictions,
- mstate->stats.cache_overflows,
+ mstate->stats.cache_overflows);
+ if (es->machine)
+ appendStringInfo(es->str, " Memory Usage: " INT64_FORMAT "kB",
memPeakKb);
+
+ appendStringInfoChar(es->str, '\n');
}
}
@@ -3262,13 +3284,16 @@ show_hashagg_info(AggState *aggstate, ExplainState *es)
Agg *agg = (Agg *) aggstate->ss.ps.plan;
int64 memPeakKb = (aggstate->hash_mem_peak + 1023) / 1024;
+ /* XXX: there's nothing portable we can show here ? */
+ if (!es->machine)
+ return;
+
if (agg->aggstrategy != AGG_HASHED &&
agg->aggstrategy != AGG_MIXED)
return;
if (es->format != EXPLAIN_FORMAT_TEXT)
{
-
if (es->costs)
ExplainPropertyInteger("Planned Partitions", NULL,
aggstate->hash_planned_partitions, es);
@@ -3381,6 +3406,10 @@ show_hashagg_info(AggState *aggstate, ExplainState *es)
static void
show_tidbitmap_info(BitmapHeapScanState *planstate, ExplainState *es)
{
+ /* XXX: there's nothing portable we can show here ? */
+ if (!es->machine)
+ return;
+
if (es->format != EXPLAIN_FORMAT_TEXT)
{
ExplainPropertyInteger("Exact Heap Blocks", NULL,
diff --git a/src/include/commands/explain.h b/src/include/commands/explain.h
index ce8c458d73..931dad08cb 100644
--- a/src/include/commands/explain.h
+++ b/src/include/commands/explain.h
@@ -46,6 +46,7 @@ typedef struct ExplainState
bool timing; /* print detailed node timing */
bool summary; /* print total planning and execution timing */
bool settings; /* print modified settings */
+ bool machine; /* print memory/disk and other machine-specific output */
ExplainFormat format; /* output format */
/* state for output formatting --- not reset for each new plan tree */
int indent; /* current indentation level */
diff --git a/src/test/isolation/expected/horizons.out b/src/test/isolation/expected/horizons.out
index 4150b2dee6..ee3e495a64 100644
--- a/src/test/isolation/expected/horizons.out
+++ b/src/test/isolation/expected/horizons.out
@@ -24,7 +24,7 @@ Index Only Scan using horizons_tst_data_key on horizons_tst
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -34,7 +34,7 @@ step pruner_query:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -47,7 +47,7 @@ step pruner_delete:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -57,7 +57,7 @@ step pruner_query:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -94,7 +94,7 @@ Index Only Scan using horizons_tst_data_key on horizons_tst
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -104,7 +104,7 @@ step pruner_query:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -117,7 +117,7 @@ step pruner_delete:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -127,7 +127,7 @@ step pruner_query:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -156,7 +156,7 @@ step ll_start:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -166,7 +166,7 @@ step pruner_query:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -180,7 +180,7 @@ step pruner_delete:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -190,7 +190,7 @@ step pruner_query:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -220,7 +220,7 @@ step ll_start:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -230,7 +230,7 @@ step pruner_query:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -246,7 +246,7 @@ step pruner_vacuum:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -256,7 +256,7 @@ step pruner_query:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -285,7 +285,7 @@ step ll_start:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -295,7 +295,7 @@ step pruner_query:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -311,7 +311,7 @@ step pruner_vacuum:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
@@ -321,7 +321,7 @@ step pruner_query:
step pruner_query:
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
?column?
diff --git a/src/test/isolation/specs/horizons.spec b/src/test/isolation/specs/horizons.spec
index d5239ff228..082205d36b 100644
--- a/src/test/isolation/specs/horizons.spec
+++ b/src/test/isolation/specs/horizons.spec
@@ -82,7 +82,7 @@ step pruner_vacuum
step pruner_query
{
SELECT explain_json($$
- EXPLAIN (FORMAT json, BUFFERS, ANALYZE)
+ EXPLAIN (FORMAT json, BUFFERS, ANALYZE, machine)
SELECT * FROM horizons_tst ORDER BY data;$$)->0->'Plan'->'Heap Fetches';
}
diff --git a/src/test/regress/expected/explain.out b/src/test/regress/expected/explain.out
index 188dc7ccec..36c20e0680 100644
--- a/src/test/regress/expected/explain.out
+++ b/src/test/regress/expected/explain.out
@@ -276,6 +276,61 @@ select explain_filter('explain (buffers, format json) select * from int8_tbl i8'
]
(1 row)
+-- HashAgg
+begin;
+SET work_mem='64kB';
+select explain_filter('explain (analyze) SELECT a, COUNT(1) FROM generate_series(1,9999)a GROUP BY 1');
+ explain_filter
+----------------------------------------------------------------------------------------------------------------
+ HashAggregate (cost=N.N..N.N rows=N width=N) (actual time=N.N..N.N rows=N loops=N)
+ Group Key: a
+ Batches: N Memory Usage: NkB Disk Usage: NkB
+ -> Function Scan on generate_series a (cost=N.N..N.N rows=N width=N) (actual time=N.N..N.N rows=N loops=N)
+ Planning Time: N.N ms
+ Execution Time: N.N ms
+(6 rows)
+
+select explain_filter('explain (analyze, machine off) SELECT a, COUNT(1) FROM generate_series(1,9999)a GROUP BY 1');
+ explain_filter
+----------------------------------------------------------------------------------------------------------------
+ HashAggregate (cost=N.N..N.N rows=N width=N) (actual time=N.N..N.N rows=N loops=N)
+ Group Key: a
+ -> Function Scan on generate_series a (cost=N.N..N.N rows=N width=N) (actual time=N.N..N.N rows=N loops=N)
+ Planning Time: N.N ms
+ Execution Time: N.N ms
+(5 rows)
+
+rollback;
+-- Bitmap scan
+begin;
+SET enable_indexscan=no;
+CREATE TABLE explainbitmap AS SELECT i AS a FROM generate_series(1,999) AS i;
+ANALYZE explainbitmap;
+CREATE INDEX ON explainbitmap(a);
+select explain_filter('explain (analyze) SELECT * FROM explainbitmap WHERE a<9');
+ explain_filter
+----------------------------------------------------------------------------------------------------------------------
+ Bitmap Heap Scan on explainbitmap (cost=N.N..N.N rows=N width=N) (actual time=N.N..N.N rows=N loops=N)
+ Recheck Cond: (a < N)
+ Heap Blocks: exact=N
+ -> Bitmap Index Scan on explainbitmap_a_idx (cost=N.N..N.N rows=N width=N) (actual time=N.N..N.N rows=N loops=N)
+ Index Cond: (a < N)
+ Planning Time: N.N ms
+ Execution Time: N.N ms
+(7 rows)
+
+select explain_filter('explain (analyze, machine off) SELECT * FROM explainbitmap WHERE a<9');
+ explain_filter
+----------------------------------------------------------------------------------------------------------------------
+ Bitmap Heap Scan on explainbitmap (cost=N.N..N.N rows=N width=N) (actual time=N.N..N.N rows=N loops=N)
+ Recheck Cond: (a < N)
+ -> Bitmap Index Scan on explainbitmap_a_idx (cost=N.N..N.N rows=N width=N) (actual time=N.N..N.N rows=N loops=N)
+ Index Cond: (a < N)
+ Planning Time: N.N ms
+ Execution Time: N.N ms
+(6 rows)
+
+rollback;
-- SETTINGS option
-- We have to ignore other settings that might be imposed by the environment,
-- so printing the whole Settings field unfortunately won't do.
diff --git a/src/test/regress/expected/incremental_sort.out b/src/test/regress/expected/incremental_sort.out
index 545e301e48..56a32df536 100644
--- a/src/test/regress/expected/incremental_sort.out
+++ b/src/test/regress/expected/incremental_sort.out
@@ -542,7 +542,7 @@ select explain_analyze_without_memory('select * from (select * from t order by a
Full-sort Groups: 2 Sort Methods: top-N heapsort, quicksort Average Memory: NNkB Peak Memory: NNkB
-> Sort (actual rows=101 loops=1)
Sort Key: t.a
- Sort Method: quicksort Memory: NNkB
+ Sort Method: quicksort
-> Seq Scan on t (actual rows=1000 loops=1)
(9 rows)
@@ -745,7 +745,7 @@ select explain_analyze_without_memory('select * from (select * from t order by a
Pre-sorted Groups: 5 Sort Methods: top-N heapsort, quicksort Average Memory: NNkB Peak Memory: NNkB
-> Sort (actual rows=1000 loops=1)
Sort Key: t.a
- Sort Method: quicksort Memory: NNkB
+ Sort Method: quicksort
-> Seq Scan on t (actual rows=1000 loops=1)
(10 rows)
diff --git a/src/test/regress/expected/memoize.out b/src/test/regress/expected/memoize.out
index 00438eb1ea..1b1557ce9f 100644
--- a/src/test/regress/expected/memoize.out
+++ b/src/test/regress/expected/memoize.out
@@ -21,9 +21,7 @@ begin
end if;
ln := regexp_replace(ln, 'Evictions: 0', 'Evictions: Zero');
ln := regexp_replace(ln, 'Evictions: \d+', 'Evictions: N');
- ln := regexp_replace(ln, 'Memory Usage: \d+', 'Memory Usage: N');
- ln := regexp_replace(ln, 'Heap Fetches: \d+', 'Heap Fetches: N');
- ln := regexp_replace(ln, 'loops=\d+', 'loops=N');
+ ln := regexp_replace(ln, 'loops=\d+', 'loops=N');
return next ln;
end loop;
end;
@@ -45,11 +43,10 @@ WHERE t2.unique1 < 1000;', false);
-> Memoize (actual rows=1 loops=N)
Cache Key: t2.twenty
Cache Mode: logical
- Hits: 980 Misses: 20 Evictions: Zero Overflows: 0 Memory Usage: NkB
+ Hits: 980 Misses: 20 Evictions: Zero Overflows: 0
-> Index Only Scan using tenk1_unique1 on tenk1 t1 (actual rows=1 loops=N)
Index Cond: (unique1 = t2.twenty)
- Heap Fetches: N
-(12 rows)
+(11 rows)
-- And check we get the expected results.
SELECT COUNT(*),AVG(t1.unique1) FROM tenk1 t1
@@ -75,11 +72,10 @@ WHERE t1.unique1 < 1000;', false);
-> Memoize (actual rows=1 loops=N)
Cache Key: t1.twenty
Cache Mode: logical
- Hits: 980 Misses: 20 Evictions: Zero Overflows: 0 Memory Usage: NkB
+ Hits: 980 Misses: 20 Evictions: Zero Overflows: 0
-> Index Only Scan using tenk1_unique1 on tenk1 t2 (actual rows=1 loops=N)
Index Cond: (unique1 = t1.twenty)
- Heap Fetches: N
-(12 rows)
+(11 rows)
-- And check we get the expected results.
SELECT COUNT(*),AVG(t2.unique1) FROM tenk1 t1,
@@ -111,11 +107,10 @@ WHERE t2.unique1 < 1200;', true);
-> Memoize (actual rows=1 loops=N)
Cache Key: t2.thousand
Cache Mode: logical
- Hits: N Misses: N Evictions: N Overflows: 0 Memory Usage: NkB
+ Hits: N Misses: N Evictions: N Overflows: 0
-> Index Only Scan using tenk1_unique1 on tenk1 t1 (actual rows=1 loops=N)
Index Cond: (unique1 = t2.thousand)
- Heap Fetches: N
-(12 rows)
+(11 rows)
CREATE TABLE flt (f float);
CREATE INDEX flt_f_idx ON flt (f);
@@ -129,15 +124,13 @@ SELECT * FROM flt f1 INNER JOIN flt f2 ON f1.f = f2.f;', false);
-------------------------------------------------------------------------------
Nested Loop (actual rows=4 loops=N)
-> Index Only Scan using flt_f_idx on flt f1 (actual rows=2 loops=N)
- Heap Fetches: N
-> Memoize (actual rows=2 loops=N)
Cache Key: f1.f
Cache Mode: logical
- Hits: 1 Misses: 1 Evictions: Zero Overflows: 0 Memory Usage: NkB
+ Hits: 1 Misses: 1 Evictions: Zero Overflows: 0
-> Index Only Scan using flt_f_idx on flt f2 (actual rows=2 loops=N)
Index Cond: (f = f1.f)
- Heap Fetches: N
-(10 rows)
+(8 rows)
-- Ensure memoize operates in binary mode
SELECT explain_memoize('
@@ -146,15 +139,13 @@ SELECT * FROM flt f1 INNER JOIN flt f2 ON f1.f >= f2.f;', false);
-------------------------------------------------------------------------------
Nested Loop (actual rows=4 loops=N)
-> Index Only Scan using flt_f_idx on flt f1 (actual rows=2 loops=N)
- Heap Fetches: N
-> Memoize (actual rows=2 loops=N)
Cache Key: f1.f
Cache Mode: binary
- Hits: 0 Misses: 2 Evictions: Zero Overflows: 0 Memory Usage: NkB
+ Hits: 0 Misses: 2 Evictions: Zero Overflows: 0
-> Index Only Scan using flt_f_idx on flt f2 (actual rows=2 loops=N)
Index Cond: (f <= f1.f)
- Heap Fetches: N
-(10 rows)
+(8 rows)
DROP TABLE flt;
-- Exercise Memoize in binary mode with a large fixed width type and a
@@ -176,7 +167,7 @@ SELECT * FROM strtest s1 INNER JOIN strtest s2 ON s1.n >= s2.n;', false);
-> Memoize (actual rows=4 loops=N)
Cache Key: s1.n
Cache Mode: binary
- Hits: 3 Misses: 3 Evictions: Zero Overflows: 0 Memory Usage: NkB
+ Hits: 3 Misses: 3 Evictions: Zero Overflows: 0
-> Index Scan using strtest_n_idx on strtest s2 (actual rows=4 loops=N)
Index Cond: (n <= s1.n)
(8 rows)
@@ -191,7 +182,7 @@ SELECT * FROM strtest s1 INNER JOIN strtest s2 ON s1.t >= s2.t;', false);
-> Memoize (actual rows=4 loops=N)
Cache Key: s1.t
Cache Mode: binary
- Hits: 3 Misses: 3 Evictions: Zero Overflows: 0 Memory Usage: NkB
+ Hits: 3 Misses: 3 Evictions: Zero Overflows: 0
-> Index Scan using strtest_t_idx on strtest s2 (actual rows=4 loops=N)
Index Cond: (t <= s1.t)
(8 rows)
diff --git a/src/test/regress/expected/partition_prune.out b/src/test/regress/expected/partition_prune.out
index 7555764c77..cabadd48b8 100644
--- a/src/test/regress/expected/partition_prune.out
+++ b/src/test/regress/expected/partition_prune.out
@@ -2479,7 +2479,6 @@ update ab_a1 set b = 3 from ab where ab.a = 1 and ab.a = ab_a1.a;
Index Cond: (a = 1)
-> Bitmap Heap Scan on ab_a1_b2 ab_a1_2 (actual rows=1 loops=1)
Recheck Cond: (a = 1)
- Heap Blocks: exact=1
-> Bitmap Index Scan on ab_a1_b2_a_idx (actual rows=1 loops=1)
Index Cond: (a = 1)
-> Bitmap Heap Scan on ab_a1_b3 ab_a1_3 (actual rows=0 loops=1)
@@ -2494,14 +2493,13 @@ update ab_a1 set b = 3 from ab where ab.a = 1 and ab.a = ab_a1.a;
Index Cond: (a = 1)
-> Bitmap Heap Scan on ab_a1_b2 ab_2 (actual rows=1 loops=1)
Recheck Cond: (a = 1)
- Heap Blocks: exact=1
-> Bitmap Index Scan on ab_a1_b2_a_idx (actual rows=1 loops=1)
Index Cond: (a = 1)
-> Bitmap Heap Scan on ab_a1_b3 ab_3 (actual rows=0 loops=1)
Recheck Cond: (a = 1)
-> Bitmap Index Scan on ab_a1_b3_a_idx (actual rows=1 loops=1)
Index Cond: (a = 1)
-(34 rows)
+(32 rows)
table ab;
a | b
diff --git a/src/test/regress/expected/select_parallel.out b/src/test/regress/expected/select_parallel.out
index 4ea1aa7dfd..9f36c62741 100644
--- a/src/test/regress/expected/select_parallel.out
+++ b/src/test/regress/expected/select_parallel.out
@@ -562,24 +562,11 @@ explain (analyze, timing off, summary off, costs off)
alter table tenk2 reset (parallel_workers);
reset work_mem;
-create function explain_parallel_sort_stats() returns setof text
-language plpgsql as
-$$
-declare ln text;
-begin
- for ln in
- explain (analyze, timing off, summary off, costs off)
- select * from
+explain (analyze)
+select * from
(select ten from tenk1 where ten < 100 order by ten) ss
- right join (values (1),(2),(3)) v(x) on true
- loop
- ln := regexp_replace(ln, 'Memory: \S*', 'Memory: xxx');
- return next ln;
- end loop;
-end;
-$$;
-select * from explain_parallel_sort_stats();
- explain_parallel_sort_stats
+ right join (values (1),(2),(3)) v(x) on true;
+ QUERY PLAN
--------------------------------------------------------------------------
Nested Loop Left Join (actual rows=30000 loops=1)
-> Values Scan on "*VALUES*" (actual rows=3 loops=1)
@@ -588,11 +575,11 @@ select * from explain_parallel_sort_stats();
Workers Launched: 4
-> Sort (actual rows=2000 loops=15)
Sort Key: tenk1.ten
- Sort Method: quicksort Memory: xxx
- Worker 0: Sort Method: quicksort Memory: xxx
- Worker 1: Sort Method: quicksort Memory: xxx
- Worker 2: Sort Method: quicksort Memory: xxx
- Worker 3: Sort Method: quicksort Memory: xxx
+ Sort Method: quicksort
+ Worker 0: Sort Method: quicksort
+ Worker 1: Sort Method: quicksort
+ Worker 2: Sort Method: quicksort
+ Worker 3: Sort Method: quicksort
-> Parallel Seq Scan on tenk1 (actual rows=2000 loops=15)
Filter: (ten < 100)
(14 rows)
@@ -603,7 +590,6 @@ reset enable_mergejoin;
reset enable_material;
reset effective_io_concurrency;
drop table bmscantest;
-drop function explain_parallel_sort_stats();
-- test parallel merge join path.
set enable_hashjoin to off;
set enable_nestloop to off;
diff --git a/src/test/regress/expected/subselect.out b/src/test/regress/expected/subselect.out
index 45c75eecc5..527f04e122 100644
--- a/src/test/regress/expected/subselect.out
+++ b/src/test/regress/expected/subselect.out
@@ -1550,27 +1550,15 @@ insert into sq_limit values
(6, 2, 2),
(7, 3, 3),
(8, 4, 4);
-create function explain_sq_limit() returns setof text language plpgsql as
-$$
-declare ln text;
-begin
- for ln in
- explain (analyze, summary off, timing off, costs off)
- select * from (select pk,c2 from sq_limit order by c1,pk) as x limit 3
- loop
- ln := regexp_replace(ln, 'Memory: \S*', 'Memory: xxx');
- return next ln;
- end loop;
-end;
-$$;
-select * from explain_sq_limit();
- explain_sq_limit
+explain (analyze)
+ select * from (select pk,c2 from sq_limit order by c1,pk) as x limit 3;
+ QUERY PLAN
----------------------------------------------------------------
Limit (actual rows=3 loops=1)
-> Subquery Scan on x (actual rows=3 loops=1)
-> Sort (actual rows=3 loops=1)
Sort Key: sq_limit.c1, sq_limit.pk
- Sort Method: top-N heapsort Memory: xxx
+ Sort Method: top-N heapsort
-> Seq Scan on sq_limit (actual rows=8 loops=1)
(6 rows)
@@ -1582,7 +1570,6 @@ select * from (select pk,c2 from sq_limit order by c1,pk) as x limit 3;
2 | 2
(3 rows)
-drop function explain_sq_limit();
drop table sq_limit;
--
-- Ensure that backward scan direction isn't propagated into
diff --git a/src/test/regress/sql/explain.sql b/src/test/regress/sql/explain.sql
index f91d1004d5..c38857d346 100644
--- a/src/test/regress/sql/explain.sql
+++ b/src/test/regress/sql/explain.sql
@@ -72,6 +72,23 @@ select explain_filter('explain (analyze, buffers, format yaml) select * from int
select explain_filter('explain (buffers, format text) select * from int8_tbl i8');
select explain_filter('explain (buffers, format json) select * from int8_tbl i8');
+-- HashAgg
+begin;
+SET work_mem='64kB';
+select explain_filter('explain (analyze) SELECT a, COUNT(1) FROM generate_series(1,9999)a GROUP BY 1');
+select explain_filter('explain (analyze, machine off) SELECT a, COUNT(1) FROM generate_series(1,9999)a GROUP BY 1');
+rollback;
+
+-- Bitmap scan
+begin;
+SET enable_indexscan=no;
+CREATE TABLE explainbitmap AS SELECT i AS a FROM generate_series(1,999) AS i;
+ANALYZE explainbitmap;
+CREATE INDEX ON explainbitmap(a);
+select explain_filter('explain (analyze) SELECT * FROM explainbitmap WHERE a<9');
+select explain_filter('explain (analyze, machine off) SELECT * FROM explainbitmap WHERE a<9');
+rollback;
+
-- SETTINGS option
-- We have to ignore other settings that might be imposed by the environment,
-- so printing the whole Settings field unfortunately won't do.
diff --git a/src/test/regress/sql/memoize.sql b/src/test/regress/sql/memoize.sql
index 0979bcdf76..5d3e37f92d 100644
--- a/src/test/regress/sql/memoize.sql
+++ b/src/test/regress/sql/memoize.sql
@@ -22,9 +22,7 @@ begin
end if;
ln := regexp_replace(ln, 'Evictions: 0', 'Evictions: Zero');
ln := regexp_replace(ln, 'Evictions: \d+', 'Evictions: N');
- ln := regexp_replace(ln, 'Memory Usage: \d+', 'Memory Usage: N');
- ln := regexp_replace(ln, 'Heap Fetches: \d+', 'Heap Fetches: N');
- ln := regexp_replace(ln, 'loops=\d+', 'loops=N');
+ ln := regexp_replace(ln, 'loops=\d+', 'loops=N');
return next ln;
end loop;
end;
diff --git a/src/test/regress/sql/select_parallel.sql b/src/test/regress/sql/select_parallel.sql
index f924731248..fc58616123 100644
--- a/src/test/regress/sql/select_parallel.sql
+++ b/src/test/regress/sql/select_parallel.sql
@@ -221,23 +221,10 @@ explain (analyze, timing off, summary off, costs off)
alter table tenk2 reset (parallel_workers);
reset work_mem;
-create function explain_parallel_sort_stats() returns setof text
-language plpgsql as
-$$
-declare ln text;
-begin
- for ln in
- explain (analyze, timing off, summary off, costs off)
- select * from
+explain (analyze)
+select * from
(select ten from tenk1 where ten < 100 order by ten) ss
- right join (values (1),(2),(3)) v(x) on true
- loop
- ln := regexp_replace(ln, 'Memory: \S*', 'Memory: xxx');
- return next ln;
- end loop;
-end;
-$$;
-select * from explain_parallel_sort_stats();
+ right join (values (1),(2),(3)) v(x) on true;
reset enable_indexscan;
reset enable_hashjoin;
@@ -245,7 +232,6 @@ reset enable_mergejoin;
reset enable_material;
reset effective_io_concurrency;
drop table bmscantest;
-drop function explain_parallel_sort_stats();
-- test parallel merge join path.
set enable_hashjoin to off;
diff --git a/src/test/regress/sql/subselect.sql b/src/test/regress/sql/subselect.sql
index 94ba91f5bb..2311c9c0ed 100644
--- a/src/test/regress/sql/subselect.sql
+++ b/src/test/regress/sql/subselect.sql
@@ -807,26 +807,11 @@ insert into sq_limit values
(7, 3, 3),
(8, 4, 4);
-create function explain_sq_limit() returns setof text language plpgsql as
-$$
-declare ln text;
-begin
- for ln in
- explain (analyze, summary off, timing off, costs off)
- select * from (select pk,c2 from sq_limit order by c1,pk) as x limit 3
- loop
- ln := regexp_replace(ln, 'Memory: \S*', 'Memory: xxx');
- return next ln;
- end loop;
-end;
-$$;
-
-select * from explain_sq_limit();
+explain (analyze)
+ select * from (select pk,c2 from sq_limit order by c1,pk) as x limit 3;
select * from (select pk,c2 from sq_limit order by c1,pk) as x limit 3;
-drop function explain_sq_limit();
-
drop table sq_limit;
--
--
2.17.1
[text/x-diff] 0005-f-Rows-removed-by-filter.patch (18.4K, ../../[email protected]/6-0005-f-Rows-removed-by-filter.patch)
download | inline diff:
From a33df91a219c74318bb355369bfa7da5a25d3924 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Tue, 16 Nov 2021 11:22:40 -0600
Subject: [PATCH 5/6] f!Rows removed by filter
This cleans one more kludge in partition_prune, but drags in 2 more files...
---
.../postgres_fdw/expected/postgres_fdw.out | 6 ++--
src/backend/commands/explain.c | 36 +++++++++----------
src/test/regress/expected/memoize.out | 9 ++---
src/test/regress/expected/partition_prune.out | 29 +++++----------
src/test/regress/expected/select_parallel.out | 4 +--
src/test/regress/sql/partition_prune.sql | 1 -
6 files changed, 32 insertions(+), 53 deletions(-)
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index f210f91188..1d03796c54 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -10496,13 +10496,12 @@ SELECT * FROM local_tbl, async_pt WHERE local_tbl.a = async_pt.a AND local_tbl.c
Nested Loop (actual rows=1 loops=1)
-> Seq Scan on local_tbl (actual rows=1 loops=1)
Filter: (c = 'bar'::text)
- Rows Removed by Filter: 1
-> Append (actual rows=1 loops=1)
-> Async Foreign Scan on async_p1 async_pt_1 (never executed)
-> Async Foreign Scan on async_p2 async_pt_2 (actual rows=1 loops=1)
-> Seq Scan on async_p3 async_pt_3 (never executed)
Filter: (local_tbl.a = a)
-(9 rows)
+(8 rows)
SELECT * FROM local_tbl, async_pt WHERE local_tbl.a = async_pt.a AND local_tbl.c = 'bar';
a | b | c | a | b | c
@@ -10636,8 +10635,7 @@ SELECT * FROM async_pt t1 WHERE t1.b === 505 LIMIT 1;
Filter: (b === 505)
-> Seq Scan on async_p3 t1_3 (actual rows=1 loops=1)
Filter: (b === 505)
- Rows Removed by Filter: 101
-(9 rows)
+(8 rows)
SELECT * FROM async_pt t1 WHERE t1.b === 505 LIMIT 1;
a | b | c
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index 71d6921a04..1b9976255c 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -1771,7 +1771,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
show_scan_qual(((IndexScan *) plan)->indexorderbyorig,
"Order By", planstate, ancestors, es);
show_scan_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
break;
@@ -1784,7 +1784,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
show_scan_qual(((IndexOnlyScan *) plan)->indexorderby,
"Order By", planstate, ancestors, es);
show_scan_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
if (es->analyze && es->machine)
@@ -1802,7 +1802,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
show_instrumentation_count("Rows Removed by Index Recheck", 2,
planstate, es);
show_scan_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
if (es->analyze)
@@ -1820,7 +1820,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
case T_WorkTableScan:
case T_SubqueryScan:
show_scan_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
break;
@@ -1829,7 +1829,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
Gather *gather = (Gather *) plan;
show_scan_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
ExplainPropertyInteger("Workers Planned", NULL,
@@ -1857,7 +1857,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
GatherMerge *gm = (GatherMerge *) plan;
show_scan_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
ExplainPropertyInteger("Workers Planned", NULL,
@@ -1895,7 +1895,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
es->verbose, es);
}
show_scan_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
break;
@@ -1909,7 +1909,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
es->verbose, es);
}
show_scan_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
break;
@@ -1925,7 +1925,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
tidquals = list_make1(make_orclause(tidquals));
show_scan_qual(tidquals, "TID Cond", planstate, ancestors, es);
show_scan_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
}
@@ -1942,14 +1942,14 @@ ExplainNode(PlanState *planstate, List *ancestors,
tidquals = list_make1(make_andclause(tidquals));
show_scan_qual(tidquals, "TID Cond", planstate, ancestors, es);
show_scan_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
}
break;
case T_ForeignScan:
show_scan_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
show_foreignscan_info((ForeignScanState *) planstate, es);
@@ -1959,7 +1959,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
CustomScanState *css = (CustomScanState *) planstate;
show_scan_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
if (css->methods->ExplainCustomScan)
@@ -1973,7 +1973,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
show_instrumentation_count("Rows Removed by Join Filter", 1,
planstate, es);
show_upper_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 2,
planstate, es);
break;
@@ -1986,7 +1986,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
show_instrumentation_count("Rows Removed by Join Filter", 1,
planstate, es);
show_upper_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 2,
planstate, es);
break;
@@ -1999,7 +1999,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
show_instrumentation_count("Rows Removed by Join Filter", 1,
planstate, es);
show_upper_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 2,
planstate, es);
break;
@@ -2007,14 +2007,14 @@ ExplainNode(PlanState *planstate, List *ancestors,
show_agg_keys(castNode(AggState, planstate), ancestors, es);
show_upper_qual(plan->qual, "Filter", planstate, ancestors, es);
show_hashagg_info((AggState *) planstate, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
break;
case T_Group:
show_group_keys(castNode(GroupState, planstate), ancestors, es);
show_upper_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
break;
@@ -2036,7 +2036,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
show_upper_qual((List *) ((Result *) plan)->resconstantqual,
"One-Time Filter", planstate, ancestors, es);
show_upper_qual(plan->qual, "Filter", planstate, ancestors, es);
- if (plan->qual)
+ if (plan->qual && es->machine)
show_instrumentation_count("Rows Removed by Filter", 1,
planstate, es);
break;
diff --git a/src/test/regress/expected/memoize.out b/src/test/regress/expected/memoize.out
index 1b1557ce9f..7f4b73fd42 100644
--- a/src/test/regress/expected/memoize.out
+++ b/src/test/regress/expected/memoize.out
@@ -39,14 +39,13 @@ WHERE t2.unique1 < 1000;', false);
-> Nested Loop (actual rows=1000 loops=N)
-> Seq Scan on tenk1 t2 (actual rows=1000 loops=N)
Filter: (unique1 < 1000)
- Rows Removed by Filter: 9000
-> Memoize (actual rows=1 loops=N)
Cache Key: t2.twenty
Cache Mode: logical
Hits: 980 Misses: 20 Evictions: Zero Overflows: 0
-> Index Only Scan using tenk1_unique1 on tenk1 t1 (actual rows=1 loops=N)
Index Cond: (unique1 = t2.twenty)
-(11 rows)
+(10 rows)
-- And check we get the expected results.
SELECT COUNT(*),AVG(t1.unique1) FROM tenk1 t1
@@ -68,14 +67,13 @@ WHERE t1.unique1 < 1000;', false);
-> Nested Loop (actual rows=1000 loops=N)
-> Seq Scan on tenk1 t1 (actual rows=1000 loops=N)
Filter: (unique1 < 1000)
- Rows Removed by Filter: 9000
-> Memoize (actual rows=1 loops=N)
Cache Key: t1.twenty
Cache Mode: logical
Hits: 980 Misses: 20 Evictions: Zero Overflows: 0
-> Index Only Scan using tenk1_unique1 on tenk1 t2 (actual rows=1 loops=N)
Index Cond: (unique1 = t1.twenty)
-(11 rows)
+(10 rows)
-- And check we get the expected results.
SELECT COUNT(*),AVG(t2.unique1) FROM tenk1 t1,
@@ -103,14 +101,13 @@ WHERE t2.unique1 < 1200;', true);
-> Nested Loop (actual rows=1200 loops=N)
-> Seq Scan on tenk1 t2 (actual rows=1200 loops=N)
Filter: (unique1 < 1200)
- Rows Removed by Filter: 8800
-> Memoize (actual rows=1 loops=N)
Cache Key: t2.thousand
Cache Mode: logical
Hits: N Misses: N Evictions: N Overflows: 0
-> Index Only Scan using tenk1_unique1 on tenk1 t1 (actual rows=1 loops=N)
Index Cond: (unique1 = t2.thousand)
-(11 rows)
+(10 rows)
CREATE TABLE flt (f float);
CREATE INDEX flt_f_idx ON flt (f);
diff --git a/src/test/regress/expected/partition_prune.out b/src/test/regress/expected/partition_prune.out
index cabadd48b8..3576e65bc2 100644
--- a/src/test/regress/expected/partition_prune.out
+++ b/src/test/regress/expected/partition_prune.out
@@ -1922,17 +1922,13 @@ explain (analyze, costs off, summary off, timing off) select * from list_part wh
Append (actual rows=0 loops=1)
-> Seq Scan on list_part1 list_part_1 (actual rows=0 loops=1)
Filter: (a = (list_part_fn(1) + a))
- Rows Removed by Filter: 1
-> Seq Scan on list_part2 list_part_2 (actual rows=0 loops=1)
Filter: (a = (list_part_fn(1) + a))
- Rows Removed by Filter: 1
-> Seq Scan on list_part3 list_part_3 (actual rows=0 loops=1)
Filter: (a = (list_part_fn(1) + a))
- Rows Removed by Filter: 1
-> Seq Scan on list_part4 list_part_4 (actual rows=0 loops=1)
Filter: (a = (list_part_fn(1) + a))
- Rows Removed by Filter: 1
-(13 rows)
+(9 rows)
rollback;
drop table list_part;
@@ -1957,7 +1953,6 @@ begin
loop
ln := regexp_replace(ln, 'Workers Launched: \d+', 'Workers Launched: N');
ln := regexp_replace(ln, 'actual rows=\d+ loops=\d+', 'actual rows=N loops=N');
- ln := regexp_replace(ln, 'Rows Removed by Filter: \d+', 'Rows Removed by Filter: N');
return next ln;
end loop;
end;
@@ -2196,7 +2191,6 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
-> Nested Loop (actual rows=N loops=N)
-> Parallel Seq Scan on lprt_a a (actual rows=N loops=N)
Filter: (a = ANY ('{1,0,0}'::integer[]))
- Rows Removed by Filter: N
-> Append (actual rows=N loops=N)
-> Index Scan using ab_a1_b1_a_idx on ab_a1_b1 ab_1 (actual rows=N loops=N)
Index Cond: (a = a.a)
@@ -2216,7 +2210,7 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
Index Cond: (a = a.a)
-> Index Scan using ab_a3_b3_a_idx on ab_a3_b3 ab_9 (never executed)
Index Cond: (a = a.a)
-(28 rows)
+(27 rows)
delete from lprt_a where a = 1;
select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on ab.a = a.a where a.a in(1, 0, 0)');
@@ -2230,7 +2224,6 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
-> Nested Loop (actual rows=N loops=N)
-> Parallel Seq Scan on lprt_a a (actual rows=N loops=N)
Filter: (a = ANY ('{1,0,0}'::integer[]))
- Rows Removed by Filter: N
-> Append (actual rows=N loops=N)
-> Index Scan using ab_a1_b1_a_idx on ab_a1_b1 ab_1 (never executed)
Index Cond: (a = a.a)
@@ -2250,7 +2243,7 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
Index Cond: (a = a.a)
-> Index Scan using ab_a3_b3_a_idx on ab_a3_b3 ab_9 (never executed)
Index Cond: (a = a.a)
-(28 rows)
+(27 rows)
reset enable_hashjoin;
reset enable_mergejoin;
@@ -2437,14 +2430,13 @@ explain (analyze, costs off, summary off, timing off) execute ab_q6(1);
Filter: ((a = $1) AND (b = $0))
-> Seq Scan on xy_1 (actual rows=0 loops=1)
Filter: ((x = $1) AND (y = $0))
- Rows Removed by Filter: 1
-> Seq Scan on ab_a1_b1 ab_4 (never executed)
Filter: ((a = $1) AND (b = $0))
-> Seq Scan on ab_a1_b2 ab_5 (never executed)
Filter: ((a = $1) AND (b = $0))
-> Seq Scan on ab_a1_b3 ab_6 (never executed)
Filter: ((a = $1) AND (b = $0))
-(19 rows)
+(18 rows)
-- Ensure we see just the xy_1 row.
execute ab_q6(100);
@@ -3052,12 +3044,11 @@ select * from boolp where a = (select value from boolvalues where value);
InitPlan 1 (returns $0)
-> Seq Scan on boolvalues (actual rows=1 loops=1)
Filter: value
- Rows Removed by Filter: 1
-> Seq Scan on boolp_f boolp_1 (never executed)
Filter: (a = $0)
-> Seq Scan on boolp_t boolp_2 (actual rows=0 loops=1)
Filter: (a = $0)
-(9 rows)
+(8 rows)
explain (analyze, costs off, summary off, timing off)
select * from boolp where a = (select value from boolvalues where not value);
@@ -3067,12 +3058,11 @@ select * from boolp where a = (select value from boolvalues where not value);
InitPlan 1 (returns $0)
-> Seq Scan on boolvalues (actual rows=1 loops=1)
Filter: (NOT value)
- Rows Removed by Filter: 1
-> Seq Scan on boolp_f boolp_1 (actual rows=0 loops=1)
Filter: (a = $0)
-> Seq Scan on boolp_t boolp_2 (never executed)
Filter: (a = $0)
-(9 rows)
+(8 rows)
drop table boolp;
--
@@ -3096,11 +3086,9 @@ explain (analyze, costs off, summary off, timing off) execute mt_q1(15);
Subplans Removed: 1
-> Index Scan using ma_test_p2_b_idx on ma_test_p2 ma_test_1 (actual rows=1 loops=1)
Filter: ((a >= $1) AND ((a % 10) = 5))
- Rows Removed by Filter: 9
-> Index Scan using ma_test_p3_b_idx on ma_test_p3 ma_test_2 (actual rows=1 loops=1)
Filter: ((a >= $1) AND ((a % 10) = 5))
- Rows Removed by Filter: 9
-(9 rows)
+(7 rows)
execute mt_q1(15);
a
@@ -3117,8 +3105,7 @@ explain (analyze, costs off, summary off, timing off) execute mt_q1(25);
Subplans Removed: 2
-> Index Scan using ma_test_p3_b_idx on ma_test_p3 ma_test_1 (actual rows=1 loops=1)
Filter: ((a >= $1) AND ((a % 10) = 5))
- Rows Removed by Filter: 9
-(6 rows)
+(5 rows)
execute mt_q1(25);
a
diff --git a/src/test/regress/expected/select_parallel.out b/src/test/regress/expected/select_parallel.out
index 9f36c62741..02176d7e2f 100644
--- a/src/test/regress/expected/select_parallel.out
+++ b/src/test/regress/expected/select_parallel.out
@@ -551,14 +551,12 @@ explain (analyze, timing off, summary off, costs off)
-> Nested Loop (actual rows=98000 loops=1)
-> Seq Scan on tenk2 (actual rows=10 loops=1)
Filter: (thousand = 0)
- Rows Removed by Filter: 9990
-> Gather (actual rows=9800 loops=10)
Workers Planned: 4
Workers Launched: 4
-> Parallel Seq Scan on tenk1 (actual rows=1960 loops=50)
Filter: (hundred > 1)
- Rows Removed by Filter: 40
-(11 rows)
+(9 rows)
alter table tenk2 reset (parallel_workers);
reset work_mem;
diff --git a/src/test/regress/sql/partition_prune.sql b/src/test/regress/sql/partition_prune.sql
index d70bd8610c..e3938bea9c 100644
--- a/src/test/regress/sql/partition_prune.sql
+++ b/src/test/regress/sql/partition_prune.sql
@@ -463,7 +463,6 @@ begin
loop
ln := regexp_replace(ln, 'Workers Launched: \d+', 'Workers Launched: N');
ln := regexp_replace(ln, 'actual rows=\d+ loops=\d+', 'actual rows=N loops=N');
- ln := regexp_replace(ln, 'Rows Removed by Filter: \d+', 'Rows Removed by Filter: N');
return next ln;
end loop;
end;
--
2.17.1
[text/x-diff] 0006-f-Workers-Launched.patch (13.9K, ../../[email protected]/7-0006-f-Workers-Launched.patch)
download | inline diff:
From 9fceff8fc187ad057e319ad2af6b1a6179b67a79 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Tue, 16 Nov 2021 10:37:45 -0600
Subject: [PATCH 6/6] f!Workers Launched: ...
---
src/backend/commands/explain.c | 4 +-
src/test/regress/expected/partition_prune.out | 38 ++++++-------------
src/test/regress/expected/select_parallel.out | 9 ++---
src/test/regress/sql/partition_prune.sql | 1 -
4 files changed, 17 insertions(+), 35 deletions(-)
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index 1b9976255c..5083a77cb1 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -1839,7 +1839,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
if (gather->initParam)
show_eval_params(gather->initParam, es);
- if (es->analyze)
+ if (es->analyze && es->machine)
{
int nworkers;
@@ -1867,7 +1867,7 @@ ExplainNode(PlanState *planstate, List *ancestors,
if (gm->initParam)
show_eval_params(gm->initParam, es);
- if (es->analyze)
+ if (es->analyze && es->machine)
{
int nworkers;
diff --git a/src/test/regress/expected/partition_prune.out b/src/test/regress/expected/partition_prune.out
index 3576e65bc2..97c576decd 100644
--- a/src/test/regress/expected/partition_prune.out
+++ b/src/test/regress/expected/partition_prune.out
@@ -1951,7 +1951,6 @@ begin
execute format('explain (analyze, costs off, summary off, timing off) %s',
$1)
loop
- ln := regexp_replace(ln, 'Workers Launched: \d+', 'Workers Launched: N');
ln := regexp_replace(ln, 'actual rows=\d+ loops=\d+', 'actual rows=N loops=N');
return next ln;
end loop;
@@ -1970,7 +1969,6 @@ select explain_parallel_append('execute ab_q4 (2, 2)');
Finalize Aggregate (actual rows=N loops=N)
-> Gather (actual rows=N loops=N)
Workers Planned: 2
- Workers Launched: N
-> Partial Aggregate (actual rows=N loops=N)
-> Parallel Append (actual rows=N loops=N)
Subplans Removed: 6
@@ -1980,7 +1978,7 @@ select explain_parallel_append('execute ab_q4 (2, 2)');
Filter: ((a >= $1) AND (a <= $2) AND (b < 4))
-> Parallel Seq Scan on ab_a2_b3 ab_3 (actual rows=N loops=N)
Filter: ((a >= $1) AND (a <= $2) AND (b < 4))
-(13 rows)
+(12 rows)
-- Test run-time pruning with IN lists.
prepare ab_q5 (int, int, int) as
@@ -1991,7 +1989,6 @@ select explain_parallel_append('execute ab_q5 (1, 1, 1)');
Finalize Aggregate (actual rows=N loops=N)
-> Gather (actual rows=N loops=N)
Workers Planned: 2
- Workers Launched: N
-> Partial Aggregate (actual rows=N loops=N)
-> Parallel Append (actual rows=N loops=N)
Subplans Removed: 6
@@ -2001,7 +1998,7 @@ select explain_parallel_append('execute ab_q5 (1, 1, 1)');
Filter: ((b < 4) AND (a = ANY (ARRAY[$1, $2, $3])))
-> Parallel Seq Scan on ab_a1_b3 ab_3 (actual rows=N loops=N)
Filter: ((b < 4) AND (a = ANY (ARRAY[$1, $2, $3])))
-(13 rows)
+(12 rows)
select explain_parallel_append('execute ab_q5 (2, 3, 3)');
explain_parallel_append
@@ -2009,7 +2006,6 @@ select explain_parallel_append('execute ab_q5 (2, 3, 3)');
Finalize Aggregate (actual rows=N loops=N)
-> Gather (actual rows=N loops=N)
Workers Planned: 2
- Workers Launched: N
-> Partial Aggregate (actual rows=N loops=N)
-> Parallel Append (actual rows=N loops=N)
Subplans Removed: 3
@@ -2025,7 +2021,7 @@ select explain_parallel_append('execute ab_q5 (2, 3, 3)');
Filter: ((b < 4) AND (a = ANY (ARRAY[$1, $2, $3])))
-> Parallel Seq Scan on ab_a3_b3 ab_6 (actual rows=N loops=N)
Filter: ((b < 4) AND (a = ANY (ARRAY[$1, $2, $3])))
-(19 rows)
+(18 rows)
-- Try some params whose values do not belong to any partition.
select explain_parallel_append('execute ab_q5 (33, 44, 55)');
@@ -2034,11 +2030,10 @@ select explain_parallel_append('execute ab_q5 (33, 44, 55)');
Finalize Aggregate (actual rows=N loops=N)
-> Gather (actual rows=N loops=N)
Workers Planned: 2
- Workers Launched: N
-> Partial Aggregate (actual rows=N loops=N)
-> Parallel Append (actual rows=N loops=N)
Subplans Removed: 9
-(7 rows)
+(6 rows)
-- Test Parallel Append with PARAM_EXEC Params
select explain_parallel_append('select count(*) from ab where (a = (select 1) or a = (select 3)) and b = 2');
@@ -2052,7 +2047,6 @@ select explain_parallel_append('select count(*) from ab where (a = (select 1) or
-> Gather (actual rows=N loops=N)
Workers Planned: 2
Params Evaluated: $0, $1
- Workers Launched: N
-> Parallel Append (actual rows=N loops=N)
-> Parallel Seq Scan on ab_a1_b2 ab_1 (actual rows=N loops=N)
Filter: ((b = 2) AND ((a = $0) OR (a = $1)))
@@ -2060,7 +2054,7 @@ select explain_parallel_append('select count(*) from ab where (a = (select 1) or
Filter: ((b = 2) AND ((a = $0) OR (a = $1)))
-> Parallel Seq Scan on ab_a3_b2 ab_3 (actual rows=N loops=N)
Filter: ((b = 2) AND ((a = $0) OR (a = $1)))
-(16 rows)
+(15 rows)
-- Test pruning during parallel nested loop query
create table lprt_a (a int not null);
@@ -2087,7 +2081,6 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
Finalize Aggregate (actual rows=N loops=N)
-> Gather (actual rows=N loops=N)
Workers Planned: 1
- Workers Launched: N
-> Partial Aggregate (actual rows=N loops=N)
-> Nested Loop (actual rows=N loops=N)
-> Parallel Seq Scan on lprt_a a (actual rows=N loops=N)
@@ -2111,7 +2104,7 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
Index Cond: (a = a.a)
-> Index Scan using ab_a3_b3_a_idx on ab_a3_b3 ab_9 (never executed)
Index Cond: (a = a.a)
-(27 rows)
+(26 rows)
-- Ensure the same partitions are pruned when we make the nested loop
-- parameter an Expr rather than a plain Param.
@@ -2121,7 +2114,6 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
Finalize Aggregate (actual rows=N loops=N)
-> Gather (actual rows=N loops=N)
Workers Planned: 1
- Workers Launched: N
-> Partial Aggregate (actual rows=N loops=N)
-> Nested Loop (actual rows=N loops=N)
-> Parallel Seq Scan on lprt_a a (actual rows=N loops=N)
@@ -2145,7 +2137,7 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
Index Cond: (a = (a.a + 0))
-> Index Scan using ab_a3_b3_a_idx on ab_a3_b3 ab_9 (never executed)
Index Cond: (a = (a.a + 0))
-(27 rows)
+(26 rows)
insert into lprt_a values(3),(3);
select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on ab.a = a.a where a.a in(1, 0, 3)');
@@ -2154,7 +2146,6 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
Finalize Aggregate (actual rows=N loops=N)
-> Gather (actual rows=N loops=N)
Workers Planned: 1
- Workers Launched: N
-> Partial Aggregate (actual rows=N loops=N)
-> Nested Loop (actual rows=N loops=N)
-> Parallel Seq Scan on lprt_a a (actual rows=N loops=N)
@@ -2178,7 +2169,7 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
Index Cond: (a = a.a)
-> Index Scan using ab_a3_b3_a_idx on ab_a3_b3 ab_9 (actual rows=N loops=N)
Index Cond: (a = a.a)
-(27 rows)
+(26 rows)
select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on ab.a = a.a where a.a in(1, 0, 0)');
explain_parallel_append
@@ -2186,7 +2177,6 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
Finalize Aggregate (actual rows=N loops=N)
-> Gather (actual rows=N loops=N)
Workers Planned: 1
- Workers Launched: N
-> Partial Aggregate (actual rows=N loops=N)
-> Nested Loop (actual rows=N loops=N)
-> Parallel Seq Scan on lprt_a a (actual rows=N loops=N)
@@ -2210,7 +2200,7 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
Index Cond: (a = a.a)
-> Index Scan using ab_a3_b3_a_idx on ab_a3_b3 ab_9 (never executed)
Index Cond: (a = a.a)
-(27 rows)
+(26 rows)
delete from lprt_a where a = 1;
select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on ab.a = a.a where a.a in(1, 0, 0)');
@@ -2219,7 +2209,6 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
Finalize Aggregate (actual rows=N loops=N)
-> Gather (actual rows=N loops=N)
Workers Planned: 1
- Workers Launched: N
-> Partial Aggregate (actual rows=N loops=N)
-> Nested Loop (actual rows=N loops=N)
-> Parallel Seq Scan on lprt_a a (actual rows=N loops=N)
@@ -2243,7 +2232,7 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
Index Cond: (a = a.a)
-> Index Scan using ab_a3_b3_a_idx on ab_a3_b3 ab_9 (never executed)
Index Cond: (a = a.a)
-(27 rows)
+(26 rows)
reset enable_hashjoin;
reset enable_mergejoin;
@@ -3664,7 +3653,6 @@ select explain_parallel_append('select * from listp where a = (select 1);');
Gather (actual rows=N loops=N)
Workers Planned: 2
Params Evaluated: $0
- Workers Launched: N
InitPlan 1 (returns $0)
-> Result (actual rows=N loops=N)
-> Parallel Append (actual rows=N loops=N)
@@ -3672,7 +3660,7 @@ select explain_parallel_append('select * from listp where a = (select 1);');
Filter: (a = $0)
-> Parallel Seq Scan on listp_12_2 listp_2 (never executed)
Filter: (a = $0)
-(11 rows)
+(10 rows)
-- Like the above but throw some more complexity at the planner by adding
-- a UNION ALL. We expect both sides of the union not to scan the
@@ -3687,7 +3675,6 @@ select * from listp where a = (select 2);');
-> Gather (actual rows=N loops=N)
Workers Planned: 2
Params Evaluated: $0
- Workers Launched: N
InitPlan 1 (returns $0)
-> Result (actual rows=N loops=N)
-> Parallel Append (actual rows=N loops=N)
@@ -3698,7 +3685,6 @@ select * from listp where a = (select 2);');
-> Gather (actual rows=N loops=N)
Workers Planned: 2
Params Evaluated: $1
- Workers Launched: N
InitPlan 2 (returns $1)
-> Result (actual rows=N loops=N)
-> Parallel Append (actual rows=N loops=N)
@@ -3706,7 +3692,7 @@ select * from listp where a = (select 2);');
Filter: (a = $1)
-> Parallel Seq Scan on listp_12_2 listp_5 (actual rows=N loops=N)
Filter: (a = $1)
-(23 rows)
+(21 rows)
drop table listp;
reset parallel_tuple_cost;
diff --git a/src/test/regress/expected/select_parallel.out b/src/test/regress/expected/select_parallel.out
index 02176d7e2f..4c2e83c763 100644
--- a/src/test/regress/expected/select_parallel.out
+++ b/src/test/regress/expected/select_parallel.out
@@ -553,10 +553,9 @@ explain (analyze, timing off, summary off, costs off)
Filter: (thousand = 0)
-> Gather (actual rows=9800 loops=10)
Workers Planned: 4
- Workers Launched: 4
-> Parallel Seq Scan on tenk1 (actual rows=1960 loops=50)
Filter: (hundred > 1)
-(9 rows)
+(8 rows)
alter table tenk2 reset (parallel_workers);
reset work_mem;
@@ -570,7 +569,6 @@ select * from
-> Values Scan on "*VALUES*" (actual rows=3 loops=1)
-> Gather Merge (actual rows=10000 loops=3)
Workers Planned: 4
- Workers Launched: 4
-> Sort (actual rows=2000 loops=15)
Sort Key: tenk1.ten
Sort Method: quicksort
@@ -580,7 +578,7 @@ select * from
Worker 3: Sort Method: quicksort
-> Parallel Seq Scan on tenk1 (actual rows=2000 loops=15)
Filter: (ten < 100)
-(14 rows)
+(13 rows)
reset enable_indexscan;
reset enable_hashjoin;
@@ -1032,9 +1030,8 @@ EXPLAIN (analyze, timing off, summary off, costs off) SELECT * FROM tenk1;
-------------------------------------------------------------
Gather (actual rows=10000 loops=1)
Workers Planned: 4
- Workers Launched: 4
-> Parallel Seq Scan on tenk1 (actual rows=2000 loops=5)
-(4 rows)
+(3 rows)
ROLLBACK TO SAVEPOINT settings;
-- provoke error in worker
diff --git a/src/test/regress/sql/partition_prune.sql b/src/test/regress/sql/partition_prune.sql
index e3938bea9c..24908a91f6 100644
--- a/src/test/regress/sql/partition_prune.sql
+++ b/src/test/regress/sql/partition_prune.sql
@@ -461,7 +461,6 @@ begin
execute format('explain (analyze, costs off, summary off, timing off) %s',
$1)
loop
- ln := regexp_replace(ln, 'Workers Launched: \d+', 'Workers Launched: N');
ln := regexp_replace(ln, 'actual rows=\d+ loops=\d+', 'actual rows=N loops=N');
return next ln;
end loop;
--
2.17.1
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE))
@ 2022-03-30 19:21 Justin Pryzby <[email protected]>
parent: Justin Pryzby <[email protected]>
0 siblings, 1 reply; 14+ messages in thread
From: Justin Pryzby @ 2022-03-30 19:21 UTC (permalink / raw)
To: Nikolay Samokhvalov <[email protected]>; +Cc: [email protected]
On Sat, Feb 26, 2022 at 03:07:20PM -0600, Justin Pryzby wrote:
> Rebased over ebf6c5249b7db525e59563fb149642665c88f747.
> It looks like that patch handles only query_id, and this patch also tries to
> handle a bunch of other stuff.
>
> If it's helpful, feel free to kick this patch to a future CF.
Rebased over MERGE.
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE))
@ 2022-07-07 06:49 Justin Pryzby <[email protected]>
parent: Justin Pryzby <[email protected]>
0 siblings, 0 replies; 14+ messages in thread
From: Justin Pryzby @ 2022-07-07 06:49 UTC (permalink / raw)
To: Nikolay Samokhvalov <[email protected]>; +Cc: [email protected]
@cfbot: rebased
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE))
@ 2022-07-26 22:38 David G. Johnston <[email protected]>
parent: Justin Pryzby <[email protected]>
1 sibling, 1 reply; 14+ messages in thread
From: David G. Johnston @ 2022-07-26 22:38 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Nikolay Samokhvalov <[email protected]>; PostgreSQL Hackers <[email protected]>
--00000000000017091c05e4bcf883
Content-Type: text/plain; charset="UTF-8"
On Mon, Jan 24, 2022 at 9:54 AM Justin Pryzby <[email protected]> wrote:
> I'm renaming this thread for better visibility, since buffers is a small,
> optional part of the patches I sent.
>
> I made a CF entry here.
> https://commitfest.postgresql.org/36/3409/
>
> On Wed, Dec 01, 2021 at 06:58:20PM -0600, Justin Pryzby wrote:
> > On Mon, Nov 15, 2021 at 01:09:54PM -0600, Justin Pryzby wrote:
> > > Some time ago, I had a few relevant patches:
> > > 1) add explain(REGRESS) which is shorthand for (BUFFERS OFF, TIMING
> OFF, COSTS OFF, SUMMARY OFF)
> > > 2) add explain(MACHINE) which elides machine-specific output from
> explain;
> > > for example, Heap Fetches, sort spaceUsed, hash nbuckets, and
> tidbitmap stuff.
> > >
> > >
> https://www.postgresql.org/message-id/flat/[email protected]
> >
> > The attached patch series now looks like this (some minor patches are not
> > included in this list):
> >
> > 1. add GUC explain_regress, which disables TIMING, SUMMARY, COSTS;
>
> > 2. enable explain(BUFFERS) by default (but disabled by explain_regress);
>
+1
> > 3. Add explain(MACHINE) - which is disabled by explain_regress.
> > This elides various machine-specific output like Memory and Disk use.
> > Maybe it should be called something else like "QUIET" or
> "VERBOSE_MINUS_1"
> > or ??
>
INCIDENTALS ?
Aside from being 4 syllables and, for me at least, a pain to remember how
to spell, it seems to fit.
RUNTIME is probably easier, and we just need to point out the difficulty in
naming things since actual rows is still shown, and timings have their own
independent option.
>
> > The regression tests now automatically run with explain_regress=on,
> which is
> > shorthand for TIMING OFF, SUMMARY OFF, COSTS OFF, and then BUFFERS OFF.
> >
> > There's a further option called explain(MACHINE) which can be disabled
> to hide
> > portions of the output that are unstable, like Memory/Disk/Batches/
> > Heap Fetches/Heap Blocks. This allows "explain analyze" to be used more
> easily
> > in regression tests, and simplifies some existing tests that currently
> use
> > plpgsql functions to filter the output. But it doesn't handle all the
> > variations from parallel workers.
> >
> > (3) is optional, but simplifies some regression tests. The patch series
> could
> > be rephrased with (3) first.
>
I like the idea of encapsulating the logic about what to show into the
generation code instead of a post-processing routine.
I don't see any particular ordering of the three changes being most clear.
>
> > Unfortunately, "COSTS OFF" breaks postgres_fdw remote_estimate. If
> specifying
> > "COSTS ON" in postgres_fdw.c is considered to be a poor fix
Given that we have version tests scattered throughout the postgres_fdw.c
code I'd say protect it with version >= 9.0 and call it good.
But I'm assuming that we are somehow also sending over the GUC to the
remote server (which will be v16+) but I am unsure why that would be the
case. I'm done code reading for now so I'm leaving this an open
request/question.
, then I suppose
> > this patch series could do as suggested and enable buffers by default
> only when
> > ANALYZE is specified. Then postgres_fdw is not affected, and the
> > explain_regress GUC is optional: instead, we'd need to specify BUFFERS
> OFF in
> > ~100 regression tests which use EXPLAIN ANALYZE.
I'm not following the transition from the prior sentences about COSTS to
this one regarding BUFFERS.
(3) still seems useful on its
> > own.
>
It is an orthogonal feature with a different section of our user base being
catered to, so I'd agree by default.
Patch Review Comments:
The following change in the machine commit seems out-of-place; are we
fixing a bug here?
explain.c:
/* Show buffer usage in planning */
- if (bufusage)
+ if (bufusage && es->buffers)
Typo (trailing comment // without comment):
ExplainPropertyFloat("Actual Rows", NULL, rows, 0, es); //
I did a pretty thorough look-through of the locations of the various
changes and everything seems consistent with the established code in this
area (took me a bit to get to the level of comprehension though). Whether
there may be something missing I'm less able to judge.
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE))
@ 2022-09-05 11:20 Justin Pryzby <[email protected]>
parent: David G. Johnston <[email protected]>
0 siblings, 1 reply; 14+ messages in thread
From: Justin Pryzby @ 2022-09-05 11:20 UTC (permalink / raw)
To: David G. Johnston <[email protected]>; +Cc: Nikolay Samokhvalov <[email protected]>; [email protected]
On Tue, Jul 26, 2022 at 03:38:53PM -0700, David G. Johnston wrote:
> On Mon, Jan 24, 2022 at 9:54 AM Justin Pryzby <[email protected]> wrote:
> > > Unfortunately, "COSTS OFF" breaks postgres_fdw remote_estimate. If specifying
> > > "COSTS ON" in postgres_fdw.c is considered to be a poor fix , then I suppose
> > > this patch series could do as suggested and enable buffers by default only when
> > > ANALYZE is specified. Then postgres_fdw is not affected, and the
> > > explain_regress GUC is optional: instead, we'd need to specify BUFFERS OFF in
> > > ~100 regression tests which use EXPLAIN ANALYZE.
>
> I'm not following the transition from the prior sentences about COSTS to
> this one regarding BUFFERS.
In this patch, SET explain_regress=on changes to defaults to
explain(COSTS OFF), so postgres_fdw now explicitly uses COSTS ON. The
patch also changes default to (BUFFERS OFF), which is what allows
enabling buffers by default. I think I was just saying that the
enabling buffers by default depends on somehow disabling it in
regression tests (preferably not by adding 100 instances of "BUFFERS
OFF").
> The following change in the machine commit seems out-of-place; are we
> fixing a bug here?
>
> explain.c:
> /* Show buffer usage in planning */
> - if (bufusage)
> + if (bufusage && es->buffers)
I think that was an extraneous change, maybe from a conflict while
re-ordering the patches. Removed it and rebased. Thanks for looking.
--
Justin
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE))
@ 2022-10-21 02:09 Justin Pryzby <[email protected]>
parent: Justin Pryzby <[email protected]>
0 siblings, 1 reply; 14+ messages in thread
From: Justin Pryzby @ 2022-10-21 02:09 UTC (permalink / raw)
To: David G. Johnston <[email protected]>; +Cc: Nikolay Samokhvalov <[email protected]>; [email protected]
Rebased.
BTW, I think it may be that the GUC should be marked PGDLLIMPORT ?
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE))
@ 2022-10-25 13:49 Laurenz Albe <[email protected]>
parent: Justin Pryzby <[email protected]>
0 siblings, 1 reply; 14+ messages in thread
From: Laurenz Albe @ 2022-10-25 13:49 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; David G. Johnston <[email protected]>; +Cc: Nikolay Samokhvalov <[email protected]>; [email protected]
On Thu, 2022-10-20 at 21:09 -0500, Justin Pryzby wrote:
> Rebased.
I had a look at the patch set.
It applies and builds cleanly and passes the regression tests.
0001: Add GUC: explain_regress
I like the idea of the "explain_regress" GUC. That should simplify
the regression tests.
--- a/src/test/regress/pg_regress.c
+++ b/src/test/regress/pg_regress.c
@@ -625,7 +625,7 @@ initialize_environment(void)
* user's ability to set other variables through that.
*/
{
- const char *my_pgoptions = "-c intervalstyle=postgres_verbose";
+ const char *my_pgoptions = "-c intervalstyle=postgres_verbose -c explain_regress=on";
const char *old_pgoptions = getenv("PGOPTIONS");
char *new_pgoptions;
@@ -2288,6 +2288,7 @@ regression_main(int argc, char *argv[],
fputs("log_lock_waits = on\n", pg_conf);
fputs("log_temp_files = 128kB\n", pg_conf);
fputs("max_prepared_transactions = 2\n", pg_conf);
+ // fputs("explain_regress = yes\n", pg_conf);
for (sl = temp_configs; sl != NULL; sl = sl->next)
{
This code comment is a leftover and should go.
0002: exercise explain_regress
This is the promised simplification. Looks good.
0003: Make explain default to BUFFERS TRUE
Yes, please!
This patch is independent from the previous patches.
I'd like this to be applied, even if the GUC is rejected.
(I understand that that would cause more changes in the regression
test output if we changed that without introducing "explain_regress".)
The patch changes the default value of "auto_explain.log_buffers"
to "on", which makes sense. However, the documentation in
doc/src/sgml/auto-explain.sgml should be updated to reflect that.
--- a/doc/src/sgml/perform.sgml
+++ b/doc/src/sgml/perform.sgml
@@ -731,8 +731,8 @@ EXPLAIN ANALYZE SELECT * FROM polygon_tbl WHERE f1 @> polygon '(0.5,2.0)';
</para>
<para>
- <command>EXPLAIN</command> has a <literal>BUFFERS</literal> option that can be used with
- <literal>ANALYZE</literal> to get even more run time statistics:
+ <command>EXPLAIN ANALYZE</command> has a <literal>BUFFERS</literal> option which
+ provides even more run time statistics:
<screen>
EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM tenk1 WHERE unique1 < 100 AND unique2 > 9000;
This is not enough. The patch would have to update all the examples that use EXPLAIN ANALYZE.
I see two options:
1. Change the output of all the examples and move this explanation to the first example
with EXPLAIN ANALYZE:
The numbers in the <literal>Buffers:</literal> line help to identify which parts
of the query are the most I/O-intensive.
2. Change all examples that currently do *not* use BUFFERS to EXPLAIN (BUFFERS OFF).
Then you could change the example that shows BUFFERS to something like
If you do not suppress the <literal>BUFFERS</literal> option that can be used with
<command>EXPLAIN (ANALYZE)</command>, you get even more run time statistics:
0004, 0005, 0006, 0007: EXPLAIN (MACHINE)
I think it is confusing that these are included in this patch set.
EXPLAIN (MACHINE OFF) is similar to "explain_regress = on", only it goes even further:
no query ID, no Heap Fetches, no Sort details, ...
Why not add this functionality to the GUC?
0005 suppresses "rows removed by filter", but how is that machine dependent?
> BTW, I think it may be that the GUC should be marked PGDLLIMPORT ?
I think it is project policy to apply this mark wherever it is needed. Do you think
that third-party extensions might need to use this in C code?
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE))
@ 2022-10-28 22:59 Justin Pryzby <[email protected]>
parent: Laurenz Albe <[email protected]>
0 siblings, 1 reply; 14+ messages in thread
From: Justin Pryzby @ 2022-10-28 22:59 UTC (permalink / raw)
To: Laurenz Albe <[email protected]>; +Cc: David G. Johnston <[email protected]>; Nikolay Samokhvalov <[email protected]>; [email protected]
On Tue, Oct 25, 2022 at 03:49:14PM +0200, Laurenz Albe wrote:
> On Thu, 2022-10-20 at 21:09 -0500, Justin Pryzby wrote:
> > Rebased.
>
> I had a look at the patch set.
Thanks for looking
> @@ -2288,6 +2288,7 @@ regression_main(int argc, char *argv[],
> fputs("log_lock_waits = on\n", pg_conf);
> fputs("log_temp_files = 128kB\n", pg_conf);
> fputs("max_prepared_transactions = 2\n", pg_conf);
> + // fputs("explain_regress = yes\n", pg_conf);
>
> for (sl = temp_configs; sl != NULL; sl = sl->next)
> {
>
> This code comment is a leftover and should go.
No - I was wondering whether the GUC should be specified by the
environment or by the file; I think it's better by file. Then it also
needs to be in Cluster.pm.
> 0004, 0005, 0006, 0007: EXPLAIN (MACHINE)
>
> I think it is confusing that these are included in this patch set.
> EXPLAIN (MACHINE OFF) is similar to "explain_regress = on", only it goes even further:
> no query ID, no Heap Fetches, no Sort details, ...
> Why not add this functionality to the GUC?
Good question, and one I've asked myself.
explain_regress affects pre-existing uses of explain in the regression
tests, aiming to simplify current (or maybe only future) uses. And
is obviously used to allow enabling BUFFERS by default.
explain(MACHINE off) is mostly about allowing "explain ANALYZE" to be
used in regression tests. Which, right now, is rare. Maybe I shouldn't
include those patches until the earlier patches progress (or, maybe we
should just defer their discussion).
> 0005 suppresses "rows removed by filter", but how is that machine dependent?
It can vary with the number of parallel workers (see 13e8b2ee8), which
may not be "machine dependent" but the point of that patch is to allow
predictable output of EXPLAIN ANALYZE. Maybe it needs a better name (or
maybe it should be folded into the first patch).
I'm not actually sure if this patch should try to address parallel
workers at all, or (if so) if what it's doing now is adequate.
> > BTW, I think it may be that the GUC should be marked PGDLLIMPORT ?
>
> I think it is project policy to apply this mark wherever it is needed. Do you think
> that third-party extensions might need to use this in C code?
Since v15 (8ec569479), the policy is:
| On Windows, export all the server's global variables using PGDLLIMPORT markers (Robert Haas)
I'm convinced now that's what's intended here.
> This is not enough. The patch would have to update all the examples
> that use EXPLAIN ANALYZE.
Fair enough. I've left a comment to handle this later if the patch
gains any traction and 001 is progressed.
--
Justin
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE))
@ 2022-11-04 10:46 Laurenz Albe <[email protected]>
parent: Justin Pryzby <[email protected]>
0 siblings, 1 reply; 14+ messages in thread
From: Laurenz Albe @ 2022-11-04 10:46 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: David G. Johnston <[email protected]>; Nikolay Samokhvalov <[email protected]>; [email protected]
Thanks for the updated patch set!
On Fri, 2022-10-28 at 17:59 -0500, Justin Pryzby wrote:
>
> > 0004, 0005, 0006, 0007: EXPLAIN (MACHINE)
> >
> > I think it is confusing that these are included in this patch set.
> > EXPLAIN (MACHINE OFF) is similar to "explain_regress = on", only it goes even further:
> > no query ID, no Heap Fetches, no Sort details, ...
> > Why not add this functionality to the GUC?
>
> Good question, and one I've asked myself.
>
> explain_regress affects pre-existing uses of explain in the regression
> tests, aiming to simplify current (or maybe only future) uses. And
> is obviously used to allow enabling BUFFERS by default.
>
> explain(MACHINE off) is mostly about allowing "explain ANALYZE" to be
> used in regression tests. Which, right now, is rare. Maybe I shouldn't
> include those patches until the earlier patches progress (or, maybe we
> should just defer their discussion).
Essentially, "explain_regress" is to simplify the current regression tests,
and MACHINE OFF is to simplify future regression tests. That does not seem
like a meaningful distinction to me. Both are only useful for the regression
tests, and I see no need for two different knobs.
My opinion is now like this:
+1 on enabling BUFFERS by default for EXPLAIN (ANALYZE)
+0.2 on "explain_regress"
-1 on EXPLAIN (MACHINE) in addition to "explain_regress"
-0.1 on adding the MACHINE OFF functionality to "explain_regress"
"explain_regress" reduces the EXPLAIN options you need for regression tests.
This is somewhat useful, but not a big win. Also, it will make backpatching
regression tests slightly harder for the next 5 years.
Hence the +0.2 for "explain_regress".
For me, an important question is: do we really want more EXPLAIN (ANALYZE)
in the regression tests? It will probably slow the tests down, and I wonder
if there is a lot of added value, if we have to remove all the machine-specific
output anyway.
Hence the -0.1.
> > 0005 suppresses "rows removed by filter", but how is that machine dependent?
>
> It can vary with the number of parallel workers (see 13e8b2ee8), which
> may not be "machine dependent" but the point of that patch is to allow
> predictable output of EXPLAIN ANALYZE. Maybe it needs a better name (or
> maybe it should be folded into the first patch).
Now it makes sense to me. I just didn't think of that.
> I'm not actually sure if this patch should try to address parallel
> workers at all, or (if so) if what it's doing now is adequate.
>
> > > BTW, I think it may be that the GUC should be marked PGDLLIMPORT ?
> >
> > I think it is project policy to apply this mark wherever it is needed. Do you think
> > that third-party extensions might need to use this in C code?
>
> Since v15 (8ec569479), the policy is:
> > On Windows, export all the server's global variables using PGDLLIMPORT markers (Robert Haas)
>
> I'm convinced now that's what's intended here.
You convinced me too.
> > This is not enough. The patch would have to update all the examples
> > that use EXPLAIN ANALYZE.
>
> Fair enough. I've left a comment to handle this later if the patch
> gains any traction and 001 is progressed.
I agree with that.
I would really like to see 0003 go in, but it currently depends on 0001, which
I am not so sure about.
I understand that you did that so that "explain_regress" can turn off BUFFERS
and there is no extra churn in the regression tests.
Still, it would be a shame if resistance against "explain_regress" would
be a show-stopper for 0003.
If I could get my way, I'd want two separate patches: first, one to turn
BUFFERS on, and second one for "explain_regress" with its current functionality
on top of that.
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE))
@ 2022-11-05 13:44 Justin Pryzby <[email protected]>
parent: Laurenz Albe <[email protected]>
0 siblings, 1 reply; 14+ messages in thread
From: Justin Pryzby @ 2022-11-05 13:44 UTC (permalink / raw)
To: Laurenz Albe <[email protected]>; +Cc: David G. Johnston <[email protected]>; Nikolay Samokhvalov <[email protected]>; [email protected]
On Fri, Nov 04, 2022 at 11:46:03AM +0100, Laurenz Albe wrote:
> "explain_regress" reduces the EXPLAIN options you need for regression tests.
> This is somewhat useful, but not a big win. Also, it will make backpatching
> regression tests slightly harder for the next 5 years.
But it doesn't cause backpatching to be harder; if a patch is meant to
be backpatched, its author can still choose to write the old way, with
(COSTS OFF, ...)
> For me, an important question is: do we really want more EXPLAIN (ANALYZE)
> in the regression tests? It will probably slow the tests down, and I wonder
> if there is a lot of added value, if we have to remove all the machine-specific
> output anyway.
I don't intend to encourage putting lots of "explain analyze" in the
tests. Rather, this makes that a reasonable possibility rather than
something that people are discouraged from doing by its difficulty.
Using "explain analyze" still needs to be evaluated (for speed and
otherwise), same as always.
In any case, I suggested to defer review of patches 004 and beyond,
which are optional, and it distracts from the essential discussion about
001.
> I would really like to see 0003 go in, but it currently depends on 0001, which
> I am not so sure about.
> I understand that you did that so that "explain_regress" can turn off BUFFERS
> and there is no extra churn in the regression tests.
>
> Still, it would be a shame if resistance against "explain_regress" would
> be a show-stopper for 0003.
>
> If I could get my way, I'd want two separate patches: first, one to turn
> BUFFERS on, and second one for "explain_regress" with its current functionality
> on top of that.
You set the patch to "waiting on author", which indicates that there's
no need for further input or review. But, I think that's precisely
what's needed - without input from more people, what could I do to
progress the patch ? I don't think it's reasonable to put 001 first and
change thousands (actually, 1338) of regression results. If nobody
wants to discuss 001, then this patch series won't progress.
--
Justin
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE))
@ 2022-11-05 14:43 Tom Lane <[email protected]>
parent: Justin Pryzby <[email protected]>
0 siblings, 1 reply; 14+ messages in thread
From: Tom Lane @ 2022-11-05 14:43 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Laurenz Albe <[email protected]>; David G. Johnston <[email protected]>; Nikolay Samokhvalov <[email protected]>; [email protected]
Justin Pryzby <[email protected]> writes:
> You set the patch to "waiting on author", which indicates that there's
> no need for further input or review. But, I think that's precisely
> what's needed - without input from more people, what could I do to
> progress the patch ? I don't think it's reasonable to put 001 first and
> change thousands (actually, 1338) of regression results. If nobody
> wants to discuss 001, then this patch series won't progress.
Well ...
1. 0001 invents a new GUC but provides no documentation for it.
That certainly isn't committable, and it's discouraging the
discussion you seek because people have to read the whole patch
in detail to understand what is being proposed.
2. The same complaint for 0004, which labors under the additional
problem that MACHINE is one of the worst option names I've ever
seen proposed around here. It conveys *nothing* about what it does
or is good for. The fact that it's got no relationship to the GUC
name, and yet they're intimately connected, doesn't improve my
opinion of it.
3. I'm not really on board with the entire approach. Making
EXPLAIN work significantly differently for developers and test
critters than it does for everyone else seems likely to promote
confusion and hide bugs. I don't think getting rid of the need
for filter functions in test scripts is worth that.
4. I don't see how the current patches could pass under "make
installcheck" --- it looks to me like it only takes care of
applying the GUC change in installations built by pg_regress
or Cluster.pm. To make this actually work, you'd have to
add "explain_regress = on" to the ALTER DATABASE SET commands
issued for regression databases. That'd substantially increase
the problem of "it works differently than what users see", at
least for me --- I do a lot of stuff interactively in the
regression database.
5. The change in postgres_fdw.c in 0001 concerns me too.
Yeah, it will fix things if an updated postgres_fdw talks to
an updated server, but what about an older postgres_fdw?
Don't really want to see that case break; communication with
servers of different major versions is one of postgres_fdw's
main goals.
As a side note, 0001 also creates a hard break in postgres_fdw's
ability to work with pre-9.0 remote servers, which won't accept
"EXPLAIN (COSTS)". Maybe we don't care about that anymore, given
the policy we've adopted elsewhere that we won't test or support
interoperability with versions before 9.2. But we should either
make the command spelling conditional on remote version, or
officially move that goalpost.
regards, tom lane
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE))
@ 2022-12-01 18:08 Justin Pryzby <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 0 replies; 14+ messages in thread
From: Justin Pryzby @ 2022-12-01 18:08 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Laurenz Albe <[email protected]>; David G. Johnston <[email protected]>; Nikolay Samokhvalov <[email protected]>; [email protected]
On Sat, Nov 05, 2022 at 10:43:07AM -0400, Tom Lane wrote:
> Justin Pryzby <[email protected]> writes:
> > You set the patch to "waiting on author", which indicates that there's
> > no need for further input or review. But, I think that's precisely
> > what's needed - without input from more people, what could I do to
> > progress the patch ? I don't think it's reasonable to put 001 first and
> > change thousands (actually, 1338) of regression results. If nobody
> > wants to discuss 001, then this patch series won't progress.
>
> Well ...
>
> 1. 0001 invents a new GUC but provides no documentation for it.
> That certainly isn't committable, and it's discouraging the
> discussion you seek because people have to read the whole patch
> in detail to understand what is being proposed.
>
> 2. The same complaint for 0004, which labors under the additional
I suggested that the discussion be limited to the early patches (004 is
an optional, possible idea that I had, but it's evidently still causing
confusion).
> 3. I'm not really on board with the entire approach. Making
> EXPLAIN work significantly differently for developers and test
> critters than it does for everyone else seems likely to promote
> confusion and hide bugs. I don't think getting rid of the need
> for filter functions in test scripts is worth that.
I'm open to suggestions how else to move towards the goal.
Note that there's a few other things which have vaguely-similar
behavior:
HIDE_TABLEAM=on
HIDE_TOAST_COMPRESSION=on
compute_query_id = regress
Another possibility is to make a new "explain" format, like
| explain(FORMAT REGRESS, ...).
...but I don't see how that's would be less objectionable than what I've
written.
The patch record should probably be closed until someone proposes
another way to implement what's necessary to enable explain(BUFFERS) by
default.
--
Justin
^ permalink raw reply [nested|flat] 14+ messages in thread
end of thread, other threads:[~2022-12-01 18:08 UTC | newest]
Thread overview: 14+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-02-05 12:51 [PATCH v24 1/7] Built-in compression method dilipkumar <[email protected]>
2022-01-24 17:35 explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE)) Justin Pryzby <[email protected]>
2022-02-26 21:07 ` Re: explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE)) Justin Pryzby <[email protected]>
2022-03-30 19:21 ` Re: explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE)) Justin Pryzby <[email protected]>
2022-07-07 06:49 ` Re: explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE)) Justin Pryzby <[email protected]>
2022-07-26 22:38 ` Re: explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE)) David G. Johnston <[email protected]>
2022-09-05 11:20 ` Re: explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE)) Justin Pryzby <[email protected]>
2022-10-21 02:09 ` Re: explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE)) Justin Pryzby <[email protected]>
2022-10-25 13:49 ` Re: explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE)) Laurenz Albe <[email protected]>
2022-10-28 22:59 ` Re: explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE)) Justin Pryzby <[email protected]>
2022-11-04 10:46 ` Re: explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE)) Laurenz Albe <[email protected]>
2022-11-05 13:44 ` Re: explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE)) Justin Pryzby <[email protected]>
2022-11-05 14:43 ` Re: explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE)) Tom Lane <[email protected]>
2022-12-01 18:08 ` Re: explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE)) Justin Pryzby <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox