pg.ddx.io pgsql-hackers@postgresql.org mailing list archive help / color / mirror / Atom feed From: Justin Pryzby <pryzby@telsasoft.com>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Dilip Kumar <dilipbalaut@gmail.com>
Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: Alexander Korotkov <a.korotkov@postgrespro.ru>
Cc: David Steele <david@pgmasters.net>
Cc: Ildus Kurbangaliev <i.kurbangaliev@gmail.com>
Cc: Dmitry Dolgov <9erthalion6@gmail.com>
Cc: pgsql-hackers@lists.postgresql.org
Subject: Re: [HACKERS] Custom compression methods
Date: Thu, 18 Feb 2021 15:13:14 -0600
Message-ID: <20210218211314.GC14772@telsasoft.com> (raw )
In-Reply-To: <20210214202831.GM1793@telsasoft.com >
References: <CA+TgmoZzQWBEkoZEon=TXwJhwZGXPUW7ma4xxh9DaJoHB8CfWQ@mail.gmail.com >
<CAFiTN-v9Cs1MORnp-3bGZ5QBwr5v3VarSvfaDizHi1acXES5xQ@mail.gmail.com >
<CAFiTN-unUNcgU0M29bdTrGHA4agK8T7g+dpMyVMeRtPawSWhCw@mail.gmail.com >
<CAFiTN-tZgTTejUb1tU+=LFUr5LrCbLw=x8Bxz8O0ZjN+U=Cu4w@mail.gmail.com >
<CAFiTN-tyWCqoHzsKzTbk1R6PMO0pC3g-=5=giB644rLjfZkCYA@mail.gmail.com >
<CAFiTN-sRwLUbVPN7_SsXrZnzvkbB=bWDmU-wyvZD1bmRTYrBmQ@mail.gmail.com >
<CA+Tgmoa3qzewoh9JvXVm5BEA_qKewK9hfJJhyAZFXYhsx9P-1g@mail.gmail.com >
<CA+TgmoZ9Rap=NuJLSvw2u=9fibUdB2L_Cs2Wb8FD49vhiXsdpQ@mail.gmail.com >
<20210214184940.GL1793@telsasoft.com >
<20210214202831.GM1793@telsasoft.com >
@cfbot: Resending with fixes to regression tests.
I'm hoping to see check-world pass --with-lz4 on an environment other than my
own PC.
Attachments:
[text/x-diff] 0001-Disallow-compressed-data-inside-container-types.patch (6.3K, ../20210218211314.GC14772@telsasoft.com/2-0001-Disallow-compressed-data-inside-container-types.patch)
download | inline diff:
From 0fa26749845a94b1dc437c57f698b4612c931c8b Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilipkumar@localhost.localdomain>
Date: Tue, 16 Feb 2021 19:24:38 +0530
Subject: [PATCH 01/12] Disallow compressed data inside container types
Currently, we have a general rule that Datums of container types
(rows, arrays, ranges, etc) must not contain any external TOAST
pointers. But the rule for the compressed data is not defined
and no specific rule is followed e.g. while constructing the array
we decompress the comprassed field but while contructing the row
in some cases we don't decompress the compressed data whereas in
the other cases we onlle decompress while flattening the external
toast pointers. This patch make a general rule for the compressed
data i.e. we don't allow the compressed data in the container type.
---
src/backend/access/common/heaptuple.c | 28 ++++++---------------------
src/backend/access/heap/heaptoast.c | 16 ++++++++-------
src/backend/executor/execTuples.c | 11 -----------
src/include/access/heaptoast.h | 4 ++--
4 files changed, 17 insertions(+), 42 deletions(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index 24a27e387d..5c3194f96d 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -983,30 +983,14 @@ heap_expand_tuple(HeapTuple sourceTuple, TupleDesc tupleDesc)
Datum
heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc)
{
- HeapTupleHeader td;
-
- /*
- * If the tuple contains any external TOAST pointers, we have to inline
- * those fields to meet the conventions for composite-type Datums.
- */
- if (HeapTupleHasExternal(tuple))
- return toast_flatten_tuple_to_datum(tuple->t_data,
- tuple->t_len,
- tupleDesc);
-
/*
- * Fast path for easy case: just make a palloc'd copy and insert the
- * correct composite-Datum header fields (since those may not be set if
- * the given tuple came from disk, rather than from heap_form_tuple).
+ * The tuple contains compressed/external TOAST pointers, so we have
+ * to inline those fields to meet the conventions for composite-type
+ * Datums.
*/
- td = (HeapTupleHeader) palloc(tuple->t_len);
- memcpy((char *) td, (char *) tuple->t_data, tuple->t_len);
-
- HeapTupleHeaderSetDatumLength(td, tuple->t_len);
- HeapTupleHeaderSetTypeId(td, tupleDesc->tdtypeid);
- HeapTupleHeaderSetTypMod(td, tupleDesc->tdtypmod);
-
- return PointerGetDatum(td);
+ return toast_flatten_tuple_to_datum(tuple->t_data,
+ tuple->t_len,
+ tupleDesc);
}
/*
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 55bbe1d584..dd162daab9 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -549,14 +549,15 @@ toast_flatten_tuple_to_datum(HeapTupleHeader tup,
/* ----------
* toast_build_flattened_tuple -
*
- * Build a tuple containing no out-of-line toasted fields.
- * (This does not eliminate compressed or short-header datums.)
+ * Build a tuple containing no compressed/out-of-line toasted fields.
+ * (This does not eliminate short-header datums.)
*
* This is essentially just like heap_form_tuple, except that it will
- * expand any external-data pointers beforehand.
+ * expand any compressed/external-data pointers beforehand.
*
- * It's not very clear whether it would be preferable to decompress
- * in-line compressed datums while at it. For now, we don't.
+ * It is not necessary to decompress the compressed data for the
+ * correctness, but reflects an expectation that compression will be more
+ * effective if applied to the whole tuple not individual fields.
* ----------
*/
HeapTuple
@@ -589,9 +590,10 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
struct varlena *new_value;
new_value = (struct varlena *) DatumGetPointer(new_values[i]);
- if (VARATT_IS_EXTERNAL(new_value))
+ if (VARATT_IS_EXTERNAL(new_value) ||
+ VARATT_IS_COMPRESSED(new_value))
{
- new_value = detoast_external_attr(new_value);
+ new_value = detoast_attr(new_value);
new_values[i] = PointerGetDatum(new_value);
freeable_values[num_to_free++] = (Pointer) new_value;
}
diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c
index 73c35df9c9..ca7fbed576 100644
--- a/src/backend/executor/execTuples.c
+++ b/src/backend/executor/execTuples.c
@@ -2189,13 +2189,6 @@ BuildTupleFromCStrings(AttInMetadata *attinmeta, char **values)
* memory context. Beware of code that changes context between the initial
* heap_form_tuple/etc call and calling HeapTuple(Header)GetDatum.
*
- * For performance-critical callers, it could be worthwhile to take extra
- * steps to ensure that there aren't TOAST pointers in the output of
- * heap_form_tuple to begin with. It's likely however that the costs of the
- * typcache lookup and tuple disassembly/reassembly are swamped by TOAST
- * dereference costs, so that the benefits of such extra effort would be
- * minimal.
- *
* XXX it would likely be better to create wrapper functions that produce
* a composite Datum from the field values in one step. However, there's
* enough code using the existing APIs that we couldn't get rid of this
@@ -2207,10 +2200,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple)
Datum result;
TupleDesc tupDesc;
- /* No work if there are no external TOAST pointers in the tuple */
- if (!HeapTupleHeaderHasExternal(tuple))
- return PointerGetDatum(tuple);
-
/* Use the type data saved by heap_form_tuple to look up the rowtype */
tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple),
HeapTupleHeaderGetTypMod(tuple));
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h
index 8b29f1a986..c80af45012 100644
--- a/src/include/access/heaptoast.h
+++ b/src/include/access/heaptoast.h
@@ -128,8 +128,8 @@ extern Datum toast_flatten_tuple_to_datum(HeapTupleHeader tup,
/* ----------
* toast_build_flattened_tuple -
*
- * Build a tuple containing no out-of-line toasted fields.
- * (This does not eliminate compressed or short-header datums.)
+ * Build a tuple containing no compressed/out-of-line toasted fields.
+ * (This does not eliminate short-header datums.)
* ----------
*/
extern HeapTuple toast_build_flattened_tuple(TupleDesc tupleDesc,
--
2.17.0
[text/x-diff] 0002-Built-in-compression-method.patch (108.5K, ../20210218211314.GC14772@telsasoft.com/3-0002-Built-in-compression-method.patch)
download | inline diff:
From 174c64fc8c40331fee571bd4b568a72e0f67c610 Mon Sep 17 00:00:00 2001
From: dilipkumar <dilipbalaut@gmail.com>
Date: Fri, 5 Feb 2021 18:21:47 +0530
Subject: [PATCH 02/12] Built-in compression method
Add syntax allowing a compression method to be specified.
As of now there is only 2 option for build-in compression
method (pglz, lz4) which can be set while creating a table
or adding a new column. No option for altering the
compression method for an existing column.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Robert Haas and Tomas Vondra.
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
Discussions:
https://www.postgresql.org/message-id/20171213151818.75a20259@postgrespro.ru
https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40mail.gmail.com
https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail.gmail.com
https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development
https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail.gmail.com
---
configure | 116 ++++++++++
configure.ac | 17 ++
doc/src/sgml/ddl.sgml | 3 +
doc/src/sgml/ref/create_table.sgml | 25 +++
src/backend/access/Makefile | 2 +-
src/backend/access/brin/brin_tuple.c | 5 +-
src/backend/access/common/detoast.c | 103 ++++++---
src/backend/access/common/indextuple.c | 4 +-
src/backend/access/common/toast_internals.c | 63 +++---
src/backend/access/common/tupdesc.c | 8 +
src/backend/access/compression/Makefile | 17 ++
src/backend/access/compression/compress_lz4.c | 164 ++++++++++++++
.../access/compression/compress_pglz.c | 138 ++++++++++++
src/backend/access/table/toast_helper.c | 5 +-
src/backend/bootstrap/bootstrap.c | 5 +
src/backend/catalog/genbki.pl | 7 +-
src/backend/catalog/heap.c | 4 +
src/backend/catalog/index.c | 2 +
src/backend/catalog/toasting.c | 5 +
src/backend/commands/amcmds.c | 10 +
src/backend/commands/createas.c | 14 ++
src/backend/commands/matview.c | 14 ++
src/backend/commands/tablecmds.c | 111 +++++++++-
src/backend/executor/nodeModifyTable.c | 122 +++++++++++
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/nodes/nodeFuncs.c | 2 +
src/backend/nodes/outfuncs.c | 1 +
src/backend/parser/gram.y | 26 ++-
src/backend/parser/parse_utilcmd.c | 8 +
src/backend/utils/adt/pseudotypes.c | 1 +
src/backend/utils/adt/varlena.c | 42 ++++
src/bin/pg_dump/pg_backup.h | 1 +
src/bin/pg_dump/pg_dump.c | 36 +++-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/pg_dump/t/002_pg_dump.pl | 12 +-
src/bin/psql/describe.c | 28 ++-
src/include/access/compressamapi.h | 99 +++++++++
src/include/access/detoast.h | 8 +
src/include/access/toast_helper.h | 1 +
src/include/access/toast_internals.h | 19 +-
src/include/catalog/pg_am.dat | 6 +
src/include/catalog/pg_am.h | 1 +
src/include/catalog/pg_attribute.h | 8 +-
src/include/catalog/pg_proc.dat | 20 ++
src/include/catalog/pg_type.dat | 5 +
src/include/commands/defrem.h | 1 +
src/include/executor/executor.h | 4 +-
src/include/nodes/execnodes.h | 6 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 2 +
src/include/parser/kwlist.h | 1 +
src/include/pg_config.h.in | 3 +
src/include/postgres.h | 12 +-
src/test/regress/expected/compression.out | 201 ++++++++++++++++++
src/test/regress/expected/compression_1.out | 189 ++++++++++++++++
src/test/regress/expected/psql.out | 68 +++---
src/test/regress/parallel_schedule | 2 +-
src/test/regress/serial_schedule | 1 +
src/test/regress/sql/compression.sql | 85 ++++++++
src/tools/msvc/Solution.pm | 1 +
src/tools/pgindent/typedefs.list | 1 +
62 files changed, 1746 insertions(+), 123 deletions(-)
create mode 100644 src/backend/access/compression/Makefile
create mode 100644 src/backend/access/compression/compress_lz4.c
create mode 100644 src/backend/access/compression/compress_pglz.c
create mode 100644 src/include/access/compressamapi.h
create mode 100644 src/test/regress/expected/compression.out
create mode 100644 src/test/regress/expected/compression_1.out
create mode 100644 src/test/regress/sql/compression.sql
diff --git a/configure b/configure
index ce9ea36999..0895b2f326 100755
--- a/configure
+++ b/configure
@@ -699,6 +699,7 @@ with_gnu_ld
LD
LDFLAGS_SL
LDFLAGS_EX
+with_lz4
with_zlib
with_system_tzdata
with_libxslt
@@ -864,6 +865,7 @@ with_libxml
with_libxslt
with_system_tzdata
with_zlib
+with_lz4
with_gnu_ld
with_ssl
with_openssl
@@ -1569,6 +1571,7 @@ Optional Packages:
--with-system-tzdata=DIR
use system time zone data in DIR
--without-zlib do not use Zlib
+ --with-lz4 build with LZ4 support
--with-gnu-ld assume the C compiler uses GNU ld [default=no]
--with-ssl=LIB use LIB for SSL/TLS support (openssl)
--with-openssl obsolete spelling of --with-ssl=openssl
@@ -8563,6 +8566,39 @@ fi
+#
+# LZ4
+#
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5
+$as_echo_n "checking whether to build with LZ4 support... " >&6; }
+
+
+
+# Check whether --with-lz4 was given.
+if test "${with_lz4+set}" = set; then :
+ withval=$with_lz4;
+ case $withval in
+ yes)
+
+$as_echo "#define USE_LZ4 1" >>confdefs.h
+
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_lz4=no
+
+fi
+
+
+
+
#
# Assignments
#
@@ -12054,6 +12090,56 @@ fi
fi
+if test "$with_lz4" = yes ; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5
+$as_echo_n "checking for LZ4_compress in -llz4... " >&6; }
+if ${ac_cv_lib_lz4_LZ4_compress+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ ac_check_lib_save_LIBS=$LIBS
+LIBS="-llz4 $LIBS"
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+
+/* Override any GCC internal prototype to avoid an error.
+ Use char because int might match the return type of a GCC
+ builtin and then its argument prototype would still apply. */
+#ifdef __cplusplus
+extern "C"
+#endif
+char LZ4_compress ();
+int
+main ()
+{
+return LZ4_compress ();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ ac_cv_lib_lz4_LZ4_compress=yes
+else
+ ac_cv_lib_lz4_LZ4_compress=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5
+$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; }
+if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then :
+ cat >>confdefs.h <<_ACEOF
+#define HAVE_LIBLZ4 1
+_ACEOF
+
+ LIBS="-llz4 $LIBS"
+
+else
+ as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5
+fi
+
+fi
+
if test "$enable_spinlocks" = yes; then
$as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h
@@ -13320,6 +13406,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5
fi
+fi
+
+if test "$with_lz4" = yes ; then
+ for ac_header in lz4/lz4.h
+do :
+ ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default"
+if test "x$ac_cv_header_lz4_lz4_h" = xyes; then :
+ cat >>confdefs.h <<_ACEOF
+#define HAVE_LZ4_LZ4_H 1
+_ACEOF
+
+else
+ for ac_header in lz4.h
+do :
+ ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default"
+if test "x$ac_cv_header_lz4_h" = xyes; then :
+ cat >>confdefs.h <<_ACEOF
+#define HAVE_LZ4_H 1
+_ACEOF
+
+else
+ as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5
+fi
+
+done
+
+fi
+
+done
+
fi
if test "$with_gssapi" = yes ; then
diff --git a/configure.ac b/configure.ac
index 07da84d401..63940b78a0 100644
--- a/configure.ac
+++ b/configure.ac
@@ -986,6 +986,14 @@ PGAC_ARG_BOOL(with, zlib, yes,
[do not use Zlib])
AC_SUBST(with_zlib)
+#
+# LZ4
+#
+AC_MSG_CHECKING([whether to build with LZ4 support])
+PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support],
+ [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])])
+AC_SUBST(with_lz4)
+
#
# Assignments
#
@@ -1173,6 +1181,10 @@ failure. It is possible the compiler isn't looking in the proper directory.
Use --without-zlib to disable zlib support.])])
fi
+if test "$with_lz4" = yes ; then
+ AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])])
+fi
+
if test "$enable_spinlocks" = yes; then
AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.])
else
@@ -1406,6 +1418,11 @@ failure. It is possible the compiler isn't looking in the proper directory.
Use --without-zlib to disable zlib support.])])
fi
+if test "$with_lz4" = yes ; then
+ AC_CHECK_HEADERS(lz4/lz4.h, [],
+ [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])])
+fi
+
if test "$with_gssapi" = yes ; then
AC_CHECK_HEADERS(gssapi/gssapi.h, [],
[AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])])
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 1e9a4625cc..4d7ed698b9 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -3762,6 +3762,9 @@ CREATE TABLE measurement (
<productname>PostgreSQL</productname>
tables (or, possibly, foreign tables). It is possible to specify a
tablespace and storage parameters for each partition separately.
+ By default, each column in a partition inherits the compression method
+ from parent table's column, however a different compression method can be
+ set for each partition.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index 3b2b227683..3374012940 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -69,6 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI
GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] |
UNIQUE <replaceable class="parameter">index_parameters</replaceable> |
PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> |
+ COMPRESSION <replaceable class="parameter">compression_method</replaceable> |
REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ]
[ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] }
[ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -605,6 +606,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>INCLUDING COMPRESSION</literal></term>
+ <listitem>
+ <para>
+ Compression method of the columns will be copied. The default
+ behavior is to exclude compression methods, resulting in the columns
+ having the default compression method.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>INCLUDING CONSTRAINTS</literal></term>
<listitem>
@@ -981,6 +993,19 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term>
+ <listitem>
+ <para>
+ This sets the compression method for a column. The supported compression
+ methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ <literal>lz4</literal> is available only if <literal>--with-lz4</literal>
+ was used when building <productname>PostgreSQL</productname>. The default
+ is <literal>pglz</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="sql-createtable-exclude">
<term><literal>EXCLUDE [ USING <replaceable class="parameter">index_method</replaceable> ] ( <replaceable class="parameter">exclude_element</replaceable> WITH <replaceable class="parameter">operator</replaceable> [, ... ] ) <replaceable class="parameter">index_parameters</replaceable> [ WHERE ( <replaceable class="parameter">predicate</replaceable> ) ]</literal></term>
<listitem>
diff --git a/src/backend/access/Makefile b/src/backend/access/Makefile
index 0880e0a8bb..ba08bdd63e 100644
--- a/src/backend/access/Makefile
+++ b/src/backend/access/Makefile
@@ -8,7 +8,7 @@ subdir = src/backend/access
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-SUBDIRS = brin common gin gist hash heap index nbtree rmgrdesc spgist \
+SUBDIRS = brin common compression gin gist hash heap index nbtree rmgrdesc spgist \
table tablesample transam
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c
index a7eb1c9473..0ab5712c71 100644
--- a/src/backend/access/brin/brin_tuple.c
+++ b/src/backend/access/brin/brin_tuple.c
@@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple,
(atttype->typstorage == TYPSTORAGE_EXTENDED ||
atttype->typstorage == TYPSTORAGE_MAIN))
{
- Datum cvalue = toast_compress_datum(value);
+ Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc,
+ keyno);
+ Datum cvalue = toast_compress_datum(value,
+ att->attcompression);
if (DatumGetPointer(cvalue) != NULL)
{
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index d1cdbaf648..b78d49167b 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -13,6 +13,7 @@
#include "postgres.h"
+#include "access/compressamapi.h"
#include "access/detoast.h"
#include "access/table.h"
#include "access/tableam.h"
@@ -457,28 +458,78 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
}
/* ----------
- * toast_decompress_datum -
+ * toast_get_compression_oid -
*
- * Decompress a compressed version of a varlena datum
+ * Returns the Oid of the compression method stored in the compressed data. If
+ * the varlena is not compressed then returns InvalidOid.
*/
-static struct varlena *
-toast_decompress_datum(struct varlena *attr)
+Oid
+toast_get_compression_oid(struct varlena *attr)
{
- struct varlena *result;
+ if (VARATT_IS_EXTERNAL_ONDISK(attr))
+ {
+ struct varatt_external toast_pointer;
+
+ VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr);
+
+ /* fast path for non-compressed external datums */
+ if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer))
+ return InvalidOid;
+
+ /*
+ * Just fetch the toast compress header to know the compression method
+ * in the compressed data.
+ */
+ attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS);
+ }
+ else if (!VARATT_IS_COMPRESSED(attr))
+ return InvalidOid;
+
+ return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+}
+
+/* ----------
+ * toast_get_compression_handler - get the compression handler routines
+ *
+ * helper function for toast_decompress_datum and toast_decompress_datum_slice
+ */
+static inline const CompressionAmRoutine *
+toast_get_compression_handler(struct varlena *attr)
+{
+ const CompressionAmRoutine *cmroutine;
+ CompressionId cmid;
Assert(VARATT_IS_COMPRESSED(attr));
- result = (struct varlena *)
- palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ);
- SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ);
+ cmid = TOAST_COMPRESS_METHOD(attr);
- if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr),
- TOAST_COMPRESS_SIZE(attr),
- VARDATA(result),
- TOAST_COMPRESS_RAWSIZE(attr), true) < 0)
- elog(ERROR, "compressed data is corrupted");
+ /* Get the handler routines for the compression method */
+ switch (cmid)
+ {
+ case PGLZ_COMPRESSION_ID:
+ cmroutine = &pglz_compress_methods;
+ break;
+ case LZ4_COMPRESSION_ID:
+ cmroutine = &lz4_compress_methods;
+ break;
+ default:
+ elog(ERROR, "invalid compression method id %d", cmid);
+ }
- return result;
+ return cmroutine;
+}
+
+/* ----------
+ * toast_decompress_datum -
+ *
+ * Decompress a compressed version of a varlena datum
+ */
+static struct varlena *
+toast_decompress_datum(struct varlena *attr)
+{
+ const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+
+ return cmroutine->datum_decompress(attr);
}
@@ -492,22 +543,16 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- struct varlena *result;
- int32 rawsize;
-
- Assert(VARATT_IS_COMPRESSED(attr));
-
- result = (struct varlena *) palloc(slicelength + VARHDRSZ);
-
- rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr),
- VARSIZE(attr) - TOAST_COMPRESS_HDRSZ,
- VARDATA(result),
- slicelength, false);
- if (rawsize < 0)
- elog(ERROR, "compressed data is corrupted");
+ const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
- SET_VARSIZE(result, rawsize + VARHDRSZ);
- return result;
+ /*
+ * If the handler supports the slice decompression then decompress the
+ * slice otherwise decompress complete data.
+ */
+ if (cmroutine->datum_decompress_slice)
+ return cmroutine->datum_decompress_slice(attr, slicelength);
+ else
+ return cmroutine->datum_decompress(attr);
}
/* ----------
diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c
index b72a138497..1d43d5d2ff 100644
--- a/src/backend/access/common/indextuple.c
+++ b/src/backend/access/common/indextuple.c
@@ -16,6 +16,7 @@
#include "postgres.h"
+#include "access/compressamapi.h"
#include "access/detoast.h"
#include "access/heaptoast.h"
#include "access/htup_details.h"
@@ -103,7 +104,8 @@ index_form_tuple(TupleDesc tupleDescriptor,
(att->attstorage == TYPSTORAGE_EXTENDED ||
att->attstorage == TYPSTORAGE_MAIN))
{
- Datum cvalue = toast_compress_datum(untoasted_values[i]);
+ Datum cvalue = toast_compress_datum(untoasted_values[i],
+ att->attcompression);
if (DatumGetPointer(cvalue) != NULL)
{
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index 9b9da0f41b..b04c5a5eb8 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -44,46 +44,51 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid);
* ----------
*/
Datum
-toast_compress_datum(Datum value)
+toast_compress_datum(Datum value, Oid cmoid)
{
- struct varlena *tmp;
- int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value));
- int32 len;
+ struct varlena *tmp = NULL;
+ int32 valsize;
+ const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value)));
- /*
- * No point in wasting a palloc cycle if value size is out of the allowed
- * range for compression
- */
- if (valsize < PGLZ_strategy_default->min_input_size ||
- valsize > PGLZ_strategy_default->max_input_size)
- return PointerGetDatum(NULL);
+ Assert(OidIsValid(cmoid));
+
+ /* Get the handler routines for the compression method */
+ switch (cmoid)
+ {
+ case PGLZ_COMPRESSION_AM_OID:
+ cmroutine = &pglz_compress_methods;
+ break;
+ case LZ4_COMPRESSION_AM_OID:
+ cmroutine = &lz4_compress_methods;
+ break;
+ default:
+ elog(ERROR, "Invalid compression method oid %u", cmoid);
+ }
- tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- TOAST_COMPRESS_HDRSZ);
+ /* Call the actual compression function */
+ tmp = cmroutine->datum_compress((const struct varlena *) value);
+ if (!tmp)
+ return PointerGetDatum(NULL);
/*
- * We recheck the actual size even if pglz_compress() reports success,
- * because it might be satisfied with having saved as little as one byte
- * in the compressed data --- which could turn into a net loss once you
- * consider header and alignment padding. Worst case, the compressed
- * format might require three padding bytes (plus header, which is
- * included in VARSIZE(tmp)), whereas the uncompressed format would take
- * only one header byte and no padding if the value is short enough. So
- * we insist on a savings of more than 2 bytes to ensure we have a gain.
+ * We recheck the actual size even if compression reports success, because
+ * it might be satisfied with having saved as little as one byte in the
+ * compressed data --- which could turn into a net loss once you consider
+ * header and alignment padding. Worst case, the compressed format might
+ * require three padding bytes (plus header, which is included in
+ * VARSIZE(tmp)), whereas the uncompressed format would take only one
+ * header byte and no padding if the value is short enough. So we insist
+ * on a savings of more than 2 bytes to ensure we have a gain.
*/
- len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)),
- valsize,
- TOAST_COMPRESS_RAWDATA(tmp),
- PGLZ_strategy_default);
- if (len >= 0 &&
- len + TOAST_COMPRESS_HDRSZ < valsize - 2)
+ valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value));
+
+ if (VARSIZE(tmp) < valsize - 2)
{
- TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize);
- SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ);
/* successful compression */
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c
index 902f59440c..ca26fab487 100644
--- a/src/backend/access/common/tupdesc.c
+++ b/src/backend/access/common/tupdesc.c
@@ -19,6 +19,7 @@
#include "postgres.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "access/tupdesc_details.h"
#include "catalog/pg_collation.h"
@@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2)
return false;
if (attr1->attcollation != attr2->attcollation)
return false;
+ if (attr1->attcompression != attr2->attcompression)
+ return false;
/* attacl, attoptions and attfdwoptions are not even present... */
}
@@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc,
att->attstorage = typeForm->typstorage;
att->attcollation = typeForm->typcollation;
+ if (IsStorageCompressible(typeForm->typstorage))
+ att->attcompression = DefaultCompressionOid;
+ else
+ att->attcompression = InvalidOid;
+
ReleaseSysCache(tuple);
}
diff --git a/src/backend/access/compression/Makefile b/src/backend/access/compression/Makefile
new file mode 100644
index 0000000000..779f54e785
--- /dev/null
+++ b/src/backend/access/compression/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile--
+# Makefile for access/compression
+#
+# IDENTIFICATION
+# src/backend/access/compression/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/backend/access/compression
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+
+OBJS = compress_pglz.o compress_lz4.o
+
+include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
new file mode 100644
index 0000000000..1856cf7df7
--- /dev/null
+++ b/src/backend/access/compression/compress_lz4.c
@@ -0,0 +1,164 @@
+/*-------------------------------------------------------------------------
+ *
+ * compress_lz4.c
+ * lz4 compression method.
+ *
+ * Portions Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/access/compression/compress_lz4.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#ifdef HAVE_LIBLZ4
+#include <lz4.h>
+#endif
+
+#include "access/compressamapi.h"
+#include "fmgr.h"
+#include "utils/builtins.h"
+
+/*
+ * lz4_cmcompress - compression routine for lz4 compression method
+ *
+ * Compresses source into dest using the default strategy. Returns the
+ * compressed varlena, or NULL if compression fails.
+ */
+static struct varlena *
+lz4_cmcompress(const struct varlena *value)
+{
+#ifndef HAVE_LIBLZ4
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#else
+ int32 valsize;
+ int32 len;
+ int32 max_size;
+ struct varlena *tmp = NULL;
+
+ valsize = VARSIZE_ANY_EXHDR(value);
+
+ /*
+ * Get maximum size of the compressed data that lz4 compression may output
+ * and allocate the memory for holding the compressed data and the header.
+ */
+ max_size = LZ4_compressBound(valsize);
+ tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+
+ len = LZ4_compress_default(VARDATA_ANY(value),
+ (char *) tmp + VARHDRSZ_COMPRESS,
+ valsize, max_size);
+ if (len <= 0)
+ elog(ERROR, "could not compress data with lz4");
+
+ /* data is incompressible so just free the memory and return NULL */
+ if (len > valsize)
+ {
+ pfree(tmp);
+ return NULL;
+ }
+
+ SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+
+ return tmp;
+#endif
+}
+
+/*
+ * lz4_cmdecompress - decompression routine for lz4 compression method
+ *
+ * Returns the decompressed varlena.
+ */
+static struct varlena *
+lz4_cmdecompress(const struct varlena *value)
+{
+#ifndef HAVE_LIBLZ4
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#else
+ int32 rawsize;
+ struct varlena *result;
+
+ /* allocate memory for holding the uncompressed data */
+ result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
+
+ /* decompress data using lz4 routine */
+ rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ VARDATA(result),
+ VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARRAWSIZE_4B_C(value));
+ if (rawsize < 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg_internal("compressed lz4 data is corrupt")));
+
+
+ SET_VARSIZE(result, rawsize + VARHDRSZ);
+
+ return result;
+#endif
+}
+
+/*
+ * lz4_cmdecompress_slice - slice decompression routine for lz4 compression
+ *
+ * Decompresses part of the data. Returns the decompressed varlena.
+ */
+static struct varlena *
+lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+{
+#ifndef HAVE_LIBLZ4
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#else
+ int32 rawsize;
+ struct varlena *result;
+
+ /* allocate memory for holding the uncompressed data */
+ result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
+
+ /* decompress partial data using lz4 routine */
+ rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ VARDATA(result),
+ VARSIZE(value) - VARHDRSZ_COMPRESS,
+ slicelength,
+ VARRAWSIZE_4B_C(value));
+ if (rawsize < 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg_internal("compressed lz4 data is corrupt")));
+
+ SET_VARSIZE(result, rawsize + VARHDRSZ);
+
+ return result;
+#endif
+}
+
+/* ------------------------------------------------------------------------
+ * Definition of the lz4 compression access method.
+ * ------------------------------------------------------------------------
+ */
+const CompressionAmRoutine lz4_compress_methods = {
+ .type = T_CompressionAmRoutine,
+ .datum_compress = lz4_cmcompress,
+ .datum_decompress = lz4_cmdecompress,
+ .datum_decompress_slice = lz4_cmdecompress_slice
+};
+
+/* lz4 compression handler function */
+Datum
+lz4handler(PG_FUNCTION_ARGS)
+{
+#ifndef HAVE_LIBLZ4
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#else
+ PG_RETURN_POINTER(&lz4_compress_methods);
+#endif
+}
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
new file mode 100644
index 0000000000..8a4bf427cf
--- /dev/null
+++ b/src/backend/access/compression/compress_pglz.c
@@ -0,0 +1,138 @@
+/*-------------------------------------------------------------------------
+ *
+ * compress_pglz.c
+ * pglz compression method
+ *
+ * Portions Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/access/compression/compress_pglz.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "common/pg_lzcompress.h"
+
+#include "fmgr.h"
+#include "utils/builtins.h"
+
+/*
+ * pglz_cmcompress - compression routine for pglz compression method
+ *
+ * Compresses source into dest using the default strategy. Returns the
+ * compressed varlena, or NULL if compression fails.
+ */
+static struct varlena *
+pglz_cmcompress(const struct varlena *value)
+{
+ int32 valsize,
+ len;
+ struct varlena *tmp = NULL;
+
+ valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value));
+
+ /*
+ * No point in wasting a palloc cycle if value size is outside the allowed
+ * range for compression.
+ */
+ if (valsize < PGLZ_strategy_default->min_input_size ||
+ valsize > PGLZ_strategy_default->max_input_size)
+ return NULL;
+
+ /*
+ * Get maximum size of the compressed data that pglz compression may output
+ * and allocate the memory for holding the compressed data and the header.
+ */
+ tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
+ VARHDRSZ_COMPRESS);
+
+ len = pglz_compress(VARDATA_ANY(value),
+ valsize,
+ (char *) tmp + VARHDRSZ_COMPRESS,
+ NULL);
+ if (len < 0)
+ {
+ pfree(tmp);
+ return NULL;
+ }
+
+ SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+
+ return tmp;
+}
+
+/*
+ * pglz_cmdecompress - decompression routine for pglz compression method
+ *
+ * Returns the decompressed varlena.
+ */
+static struct varlena *
+pglz_cmdecompress(const struct varlena *value)
+{
+ struct varlena *result;
+ int32 rawsize;
+
+ result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
+
+ rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
+ VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARDATA(result),
+ VARRAWSIZE_4B_C(value), true);
+ if (rawsize < 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg_internal("compressed pglz data is corrupt")));
+
+ SET_VARSIZE(result, rawsize + VARHDRSZ);
+
+ return result;
+}
+
+/*
+ * pglz_decompress - slice decompression routine for pglz compression method
+ *
+ * Decompresses part of the data. Returns the decompressed varlena.
+ */
+static struct varlena *
+pglz_cmdecompress_slice(const struct varlena *value,
+ int32 slicelength)
+{
+ struct varlena *result;
+ int32 rawsize;
+
+ result = (struct varlena *) palloc(slicelength + VARHDRSZ);
+
+ rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
+ VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARDATA(result),
+ slicelength, false);
+ if (rawsize < 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg_internal("compressed pglz data is corrupt")));
+
+ SET_VARSIZE(result, rawsize + VARHDRSZ);
+
+ return result;
+}
+
+/* ------------------------------------------------------------------------
+ * Definition of the pglz compression access method.
+ * ------------------------------------------------------------------------
+ */
+const CompressionAmRoutine pglz_compress_methods = {
+ .type = T_CompressionAmRoutine,
+ .datum_compress = pglz_cmcompress,
+ .datum_decompress = pglz_cmdecompress,
+ .datum_decompress_slice = pglz_cmdecompress_slice
+};
+
+/* pglz compression handler function */
+Datum
+pglzhandler(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_POINTER(&pglz_compress_methods);
+}
diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c
index fb36151ce5..53f78f9c3e 100644
--- a/src/backend/access/table/toast_helper.c
+++ b/src/backend/access/table/toast_helper.c
@@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc)
ttc->ttc_attr[i].tai_colflags = 0;
ttc->ttc_attr[i].tai_oldexternal = NULL;
+ ttc->ttc_attr[i].tai_compression = att->attcompression;
if (ttc->ttc_oldvalues != NULL)
{
@@ -226,9 +227,11 @@ void
toast_tuple_try_compression(ToastTupleContext *ttc, int attribute)
{
Datum *value = &ttc->ttc_values[attribute];
- Datum new_value = toast_compress_datum(*value);
+ Datum new_value;
ToastAttrInfo *attr = &ttc->ttc_attr[attribute];
+ new_value = toast_compress_datum(*value, attr->tai_compression);
+
if (DatumGetPointer(new_value) != NULL)
{
/* successful compression */
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 6f615e6622..9b451eaa71 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -17,6 +17,7 @@
#include <unistd.h>
#include <signal.h>
+#include "access/compressamapi.h"
#include "access/genam.h"
#include "access/heapam.h"
#include "access/htup_details.h"
@@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness)
attrtypes[attnum]->attcacheoff = -1;
attrtypes[attnum]->atttypmod = -1;
attrtypes[attnum]->attislocal = true;
+ if (IsStorageCompressible(attrtypes[attnum]->attstorage))
+ attrtypes[attnum]->attcompression = DefaultCompressionOid;
+ else
+ attrtypes[attnum]->attcompression = InvalidOid;
if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL)
{
diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl
index b159958112..f5bb37b972 100644
--- a/src/backend/catalog/genbki.pl
+++ b/src/backend/catalog/genbki.pl
@@ -187,7 +187,9 @@ my $GenbkiNextOid = $FirstGenbkiObjectId;
my $C_COLLATION_OID =
Catalog::FindDefinedSymbolFromData($catalog_data{pg_collation},
'C_COLLATION_OID');
-
+my $PGLZ_COMPRESSION_AM_OID =
+ Catalog::FindDefinedSymbolFromData($catalog_data{pg_am},
+ 'PGLZ_COMPRESSION_AM_OID');
# Fill in pg_class.relnatts by looking at the referenced catalog's schema.
# This is ugly but there's no better place; Catalog::AddDefaultValues
@@ -906,6 +908,9 @@ sub morph_row_for_pgattr
$row->{attcollation} =
$type->{typcollation} ne '0' ? $C_COLLATION_OID : 0;
+ $row->{attcompression} =
+ $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? $PGLZ_COMPRESSION_AM_OID : 0;
+
if (defined $attr->{forcenotnull})
{
$row->{attnotnull} = 't';
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index 9abc4a1f55..b53b6b50e6 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -29,6 +29,7 @@
*/
#include "postgres.h"
+#include "access/compressamapi.h"
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/multixact.h"
@@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel,
slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal);
slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount);
slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation);
+ slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression);
if (attoptions && attoptions[natts] != (Datum) 0)
slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts];
else
@@ -1715,6 +1717,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum)
/* Unset this so no one tries to look up the generation expression */
attStruct->attgenerated = '\0';
+ attStruct->attcompression = InvalidOid;
+
/*
* Change the column name to something that isn't likely to conflict
*/
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index b4ab0b88ad..0db2a7430f 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -24,6 +24,7 @@
#include <unistd.h>
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/heapam.h"
#include "access/multixact.h"
#include "access/reloptions.h"
@@ -348,6 +349,7 @@ ConstructTupleDescriptor(Relation heapRelation,
to->attbyval = from->attbyval;
to->attstorage = from->attstorage;
to->attalign = from->attalign;
+ to->attcompression = from->attcompression;
}
else
{
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c
index d7b806020d..a549481557 100644
--- a/src/backend/catalog/toasting.c
+++ b/src/backend/catalog/toasting.c
@@ -220,6 +220,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN;
TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN;
+ /* Toast field should not be compressed */
+ TupleDescAttr(tupdesc, 0)->attcompression = InvalidOid;
+ TupleDescAttr(tupdesc, 1)->attcompression = InvalidOid;
+ TupleDescAttr(tupdesc, 2)->attcompression = InvalidOid;
+
/*
* Toast tables for regular relations go in pg_toast; those for temp
* relations go into the per-backend temp-toast-table namespace.
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index eff9535ed0..3ad4a61739 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -175,6 +175,16 @@ get_table_am_oid(const char *amname, bool missing_ok)
return get_am_type_oid(amname, AMTYPE_TABLE, missing_ok);
}
+/*
+ * get_compression_am_oid - given an access method name, look up its OID
+ * and verify it corresponds to an compression AM.
+ */
+Oid
+get_compression_am_oid(const char *amname, bool missing_ok)
+{
+ return get_am_type_oid(amname, AMTYPE_COMPRESSION, missing_ok);
+}
+
/*
* get_am_oid - given an access method name, look up its OID.
* The type is not checked.
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index dce882012e..1d17dc0d6b 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -61,6 +61,7 @@ typedef struct
CommandId output_cid; /* cmin to insert in output tuples */
int ti_options; /* table_tuple_insert performance options */
BulkInsertState bistate; /* bulk insert state */
+ TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */
} DR_intorel;
/* utility functions for CTAS definition creation */
@@ -581,6 +582,15 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self)
/* Nothing to insert if WITH NO DATA is specified. */
if (!myState->into->skipData)
{
+ /*
+ * Compare the compression method of the compressed attribute in the
+ * source tuple with target attribute and if those are different then
+ * decompress those attributes.
+ */
+ slot = CompareCompressionMethodAndDecompress(slot,
+ &myState->decompress_tuple_slot,
+ myState->rel->rd_att);
+
/*
* Note that the input slot might not be of the type of the target
* relation. That's supported by table_tuple_insert(), but slightly
@@ -619,6 +629,10 @@ intorel_shutdown(DestReceiver *self)
/* close rel, but keep lock until commit */
table_close(myState->rel, NoLock);
myState->rel = NULL;
+
+ /* release the slot used for decompressing the tuple */
+ if (myState->decompress_tuple_slot)
+ ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot);
}
/*
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index c5c25ce11d..713fc3fceb 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -56,6 +56,7 @@ typedef struct
CommandId output_cid; /* cmin to insert in output tuples */
int ti_options; /* table_tuple_insert performance options */
BulkInsertState bistate; /* bulk insert state */
+ TupleTableSlot *decompress_tuple_slot; /* to hold the decompress tuple */
} DR_transientrel;
static int matview_maintenance_depth = 0;
@@ -486,6 +487,15 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self)
{
DR_transientrel *myState = (DR_transientrel *) self;
+ /*
+ * Compare the compression method of the compressed attribute in the
+ * source tuple with target attribute and if those are different then
+ * decompress those attributes.
+ */
+ slot = CompareCompressionMethodAndDecompress(slot,
+ &myState->decompress_tuple_slot,
+ myState->transientrel->rd_att);
+
/*
* Note that the input slot might not be of the type of the target
* relation. That's supported by table_tuple_insert(), but slightly less
@@ -521,6 +531,10 @@ transientrel_shutdown(DestReceiver *self)
/* close transientrel, but keep lock until commit */
table_close(myState->transientrel, NoLock);
myState->transientrel = NULL;
+
+ /* release the slot used for decompressing the tuple */
+ if (myState->decompress_tuple_slot)
+ ExecDropSingleTupleTableSlot(myState->decompress_tuple_slot);
}
/*
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index b2457a6924..811dbc3741 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -15,6 +15,7 @@
#include "postgres.h"
#include "access/attmap.h"
+#include "access/compressamapi.h"
#include "access/genam.h"
#include "access/heapam.h"
#include "access/heapam_xlog.h"
@@ -559,7 +560,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-
+static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -853,6 +854,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
if (colDef->generated)
attr->attgenerated = colDef->generated;
+
+ /*
+ * lookup attribute's compression method and store its Oid in the
+ * attr->attcompression.
+ */
+ if (relkind == RELKIND_RELATION ||
+ relkind == RELKIND_PARTITIONED_TABLE ||
+ relkind == RELKIND_MATVIEW)
+ attr->attcompression =
+ GetAttributeCompression(attr, colDef->compression);
+ else
+ attr->attcompression = InvalidOid;
}
/*
@@ -2397,6 +2410,21 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
storage_name(def->storage),
storage_name(attribute->attstorage))));
+ /* Copy/check compression parameter */
+ if (OidIsValid(attribute->attcompression))
+ {
+ char *compression = get_am_name(attribute->attcompression);
+
+ if (!def->compression)
+ def->compression = compression;
+ else if (strcmp(def->compression, compression) != 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("column \"%s\" has a compression method conflict",
+ attributeName),
+ errdetail("%s versus %s", def->compression, compression)));
+ }
+
def->inhcount++;
/* Merge of NOT NULL constraints = OR 'em together */
def->is_not_null |= attribute->attnotnull;
@@ -2431,6 +2459,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
+ def->compression = get_am_name(attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2676,6 +2705,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
storage_name(def->storage),
storage_name(newdef->storage))));
+ /* Copy compression parameter */
+ if (!def->compression)
+ def->compression = newdef->compression;
+ else if (newdef->compression)
+ {
+ if (strcmp(def->compression, newdef->compression))
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("column \"%s\" has a compression method conflict",
+ attributeName),
+ errdetail("%s versus %s", def->compression, newdef->compression)));
+ }
+
/* Mark the column as locally defined */
def->is_local = true;
/* Merge of NOT NULL constraints = OR 'em together */
@@ -6341,6 +6383,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
attribute.attislocal = colDef->is_local;
attribute.attinhcount = colDef->inhcount;
attribute.attcollation = collOid;
+
+ /*
+ * lookup attribute's compression method and store its Oid in the
+ * attr->attcompression.
+ */
+ if (rel->rd_rel->relkind == RELKIND_RELATION ||
+ rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
+ attribute.attcompression = GetAttributeCompression(&attribute,
+ colDef->compression);
+ else
+ attribute.attcompression = InvalidOid;
+
/* attribute.attacl is handled by InsertPgAttributeTuples() */
ReleaseSysCache(typeTuple);
@@ -11860,6 +11914,22 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
ReleaseSysCache(typeTuple);
+ /* Setup attribute compression */
+ if (rel->rd_rel->relkind == RELKIND_RELATION ||
+ rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
+ {
+ /*
+ * InvalidOid for the plain/external storage otherwise default
+ * compression id.
+ */
+ if (!IsStorageCompressible(tform->typstorage))
+ attTup->attcompression = InvalidOid;
+ else if (!OidIsValid(attTup->attcompression))
+ attTup->attcompression = DefaultCompressionOid;
+ }
+ else
+ attTup->attcompression = InvalidOid;
+
CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup);
table_close(attrelation, RowExclusiveLock);
@@ -17665,3 +17735,42 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
+
+/*
+ * resolve column compression specification to an OID.
+ */
+static Oid
+GetAttributeCompression(Form_pg_attribute att, char *compression)
+{
+ char typstorage = get_typstorage(att->atttypid);
+ Oid amoid;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return DefaultCompressionOid;
+
+ amoid = get_compression_am_oid(compression, false);
+
+#ifndef HAVE_LIBLZ4
+ if (amoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+ return amoid;
+}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index 2993ba43e3..f77deee399 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -37,6 +37,8 @@
#include "postgres.h"
+#include "access/compressamapi.h"
+#include "access/detoast.h"
#include "access/heapam.h"
#include "access/htup_details.h"
#include "access/tableam.h"
@@ -2036,6 +2038,113 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate,
return slot;
}
+/*
+ * Compare the compression method of each compressed value with the
+ * compression method of the target attribute. If the compression method
+ * of the compressed value is not supported in the target attribute then
+ * decompress the value. If any of the value need to decompress then we
+ * need to store that into the new slot.
+ *
+ * The slot will hold the input slot but if any of the value need to be
+ * decompressed then the new slot will be stored into this and the old
+ * slot will be dropped.
+ */
+TupleTableSlot *
+CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
+ TupleTableSlot **outslot,
+ TupleDesc targetTupDesc)
+{
+ int i;
+ int attnum;
+ int natts = slot->tts_tupleDescriptor->natts;
+ bool decompressed_any = false;
+ bool slot_tup_deformed = false;
+ Oid cmoid;
+ TupleDesc tupleDesc = slot->tts_tupleDescriptor;
+
+ if (natts == 0)
+ return slot;
+
+ /*
+ * Loop over all the attributes in the tuple and check if any attribute is
+ * compressed and its compression method is not same as the target
+ * atrribute's compression method then decompress it.
+ */
+ for (i = 0; i < natts; i++)
+ {
+ attnum = tupleDesc->attrs[i].attnum;
+
+ if (TupleDescAttr(tupleDesc, i)->attlen == -1)
+ {
+ struct varlena *new_value;
+
+ /* fetch the values of all the attribute, if not already done */
+ if (!slot_tup_deformed)
+ {
+ slot_getallattrs(slot);
+ slot_tup_deformed = true;
+ }
+
+ /* nothing to be done, if the value is null */
+ if (slot->tts_isnull[attnum - 1])
+ continue;
+
+ new_value = (struct varlena *)
+ DatumGetPointer(slot->tts_values[attnum - 1]);
+
+ /*
+ * Get the compression method Oid stored in the toast header and
+ * compare it with the compression method of the target.
+ */
+ cmoid = toast_get_compression_oid(new_value);
+ if (OidIsValid(cmoid) &&
+ targetTupDesc->attrs[i].attcompression != cmoid)
+ {
+ new_value = detoast_attr(new_value);
+ slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
+ decompressed_any = true;
+ }
+ }
+ }
+
+ /*
+ * If we have decompressed any of the fields then we need to copy the
+ * values/null array from oldslot to the new slot and materialize the new
+ * slot.
+ */
+ if (decompressed_any)
+ {
+ TupleTableSlot *newslot = *outslot;
+
+ /*
+ * If the called has passed an invalid slot then create a new slot.
+ * Otherwise, just clear the existing tuple from the slot. This slot
+ * should be stored by the caller so that it can be reused for
+ * decompressing the subsequent tuples.
+ */
+ if (newslot == NULL)
+ newslot = MakeSingleTupleTableSlot(tupleDesc, slot->tts_ops);
+ else
+ ExecClearTuple(newslot);
+
+ /*
+ * Copy the value/null array to the new slot and materialize it,
+ * before clearing the tuple from the old slot.
+ */
+ memcpy(newslot->tts_values, slot->tts_values, natts * sizeof(Datum));
+ memcpy(newslot->tts_isnull, slot->tts_isnull, natts * sizeof(bool));
+ ExecStoreVirtualTuple(newslot);
+ ExecMaterializeSlot(newslot);
+ ExecClearTuple(slot);
+
+ *outslot = newslot;
+
+ return newslot;
+ }
+
+ return slot;
+}
+
/* ----------------------------------------------------------------
* ExecModifyTable
*
@@ -2244,6 +2353,15 @@ ExecModifyTable(PlanState *pstate)
slot = ExecFilterJunk(junkfilter, slot);
}
+ /*
+ * Compare the compression method of the compressed attribute in the
+ * source tuple with target attribute and if those are different then
+ * decompress those attributes.
+ */
+ slot = CompareCompressionMethodAndDecompress(slot,
+ &node->mt_decompress_tuple_slot,
+ resultRelInfo->ri_RelationDesc->rd_att);
+
switch (operation)
{
case CMD_INSERT:
@@ -2876,6 +2994,10 @@ ExecEndModifyTable(ModifyTableState *node)
ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot);
}
+ /* release the slot used for decompressing the tuple */
+ if (node->mt_decompress_tuple_slot)
+ ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot);
+
/*
* Free the exprcontext
*/
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 65bbc18ecb..1338e04409 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,6 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
+ COPY_STRING_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index c2d73626fc..f3592003da 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
+ COMPARE_STRING_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c
index 49357ac5c2..38226530c6 100644
--- a/src/backend/nodes/nodeFuncs.c
+++ b/src/backend/nodes/nodeFuncs.c
@@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node,
if (walker(coldef->typeName, context))
return true;
+ if (walker(coldef->compression, context))
+ return true;
if (walker(coldef->raw_default, context))
return true;
if (walker(coldef->collClause, context))
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index f5dcedf6e8..0605ef3f84 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,6 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
+ WRITE_STRING_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index dd72a9fc3c..52d92df25d 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
+%type <str> optColumnCompression
+
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
* They must be listed first so that their numeric codes do not depend on
@@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P
CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE
CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT
- COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT
- CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE
- CROSS CSV CUBE CURRENT_P
+ COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT
+ CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY
+ COST CREATE CROSS CSV CUBE CURRENT_P
CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA
CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE
@@ -3421,11 +3423,12 @@ TypedTableElement:
| TableConstraint { $$ = $1; }
;
-columnDef: ColId Typename create_generic_options ColQualList
+columnDef: ColId Typename optColumnCompression create_generic_options ColQualList
{
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
+ n->compression = $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3434,8 +3437,8 @@ columnDef: ColId Typename create_generic_options ColQualList
n->raw_default = NULL;
n->cooked_default = NULL;
n->collOid = InvalidOid;
- n->fdwoptions = $3;
- SplitColQualList($4, &n->constraints, &n->collClause,
+ n->fdwoptions = $4;
+ SplitColQualList($5, &n->constraints, &n->collClause,
yyscanner);
n->location = @1;
$$ = (Node *)n;
@@ -3480,6 +3483,14 @@ columnOptions: ColId ColQualList
}
;
+optColumnCompression:
+ COMPRESSION name
+ {
+ $$ = $2;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
| /*EMPTY*/ { $$ = NIL; }
@@ -3710,6 +3721,7 @@ TableLikeOption:
| INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; }
| STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; }
| STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; }
+ | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; }
| ALL { $$ = CREATE_TABLE_LIKE_ALL; }
;
@@ -15285,6 +15297,7 @@ unreserved_keyword:
| COMMENTS
| COMMIT
| COMMITTED
+ | COMPRESSION
| CONFIGURATION
| CONFLICT
| CONNECTION
@@ -15805,6 +15818,7 @@ bare_label_keyword:
| COMMENTS
| COMMIT
| COMMITTED
+ | COMPRESSION
| CONCURRENTLY
| CONFIGURATION
| CONFLICT
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index 75266caeb4..e262ec5bcb 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -27,6 +27,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "access/relation.h"
#include "access/reloptions.h"
@@ -1082,6 +1083,13 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
else
def->storage = 0;
+ /* Likewise, copy compression if requested */
+ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
+ && OidIsValid(attribute->attcompression))
+ def->compression = get_am_name(attribute->attcompression);
+ else
+ def->compression = NULL;
+
/* Likewise, copy comment if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) &&
(comment = GetComment(attribute->attrelid,
diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c
index c2f910d606..fe133c76c2 100644
--- a/src/backend/utils/adt/pseudotypes.c
+++ b/src/backend/utils/adt/pseudotypes.c
@@ -383,6 +383,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(language_handler);
PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler);
PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler);
PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler);
+PSEUDOTYPE_DUMMY_IO_FUNCS(compression_am_handler);
PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler);
PSEUDOTYPE_DUMMY_IO_FUNCS(internal);
PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement);
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 479ed9ae54..a35abe5f1a 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -17,9 +17,11 @@
#include <ctype.h>
#include <limits.h>
+#include "access/compressamapi.h"
#include "access/detoast.h"
#include "catalog/pg_collation.h"
#include "catalog/pg_type.h"
+#include "commands/defrem.h"
#include "common/hashfn.h"
#include "common/hex.h"
#include "common/int.h"
@@ -5299,6 +5301,46 @@ pg_column_size(PG_FUNCTION_ARGS)
PG_RETURN_INT32(result);
}
+/*
+ * Return the compression method stored in the compressed attribute. Return
+ * NULL for non varlena type or the uncompressed data.
+ */
+Datum
+pg_column_compression(PG_FUNCTION_ARGS)
+{
+ Datum value = PG_GETARG_DATUM(0);
+ int typlen;
+ Oid cmoid;
+
+ /* On first call, get the input type's typlen, and save at *fn_extra */
+ if (fcinfo->flinfo->fn_extra == NULL)
+ {
+ /* Lookup the datatype of the supplied argument */
+ Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0);
+
+ typlen = get_typlen(argtypeid);
+ if (typlen == 0) /* should not happen */
+ elog(ERROR, "cache lookup failed for type %u", argtypeid);
+
+ fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt,
+ sizeof(int));
+ *((int *) fcinfo->flinfo->fn_extra) = typlen;
+ }
+ else
+ typlen = *((int *) fcinfo->flinfo->fn_extra);
+
+ if (typlen != -1)
+ PG_RETURN_NULL();
+
+ cmoid =
+ toast_get_compression_oid((struct varlena *) DatumGetPointer(value));
+
+ if (!OidIsValid(cmoid))
+ PG_RETURN_NULL();
+ else
+ PG_RETURN_TEXT_P(cstring_to_text(get_am_name(cmoid)));
+}
+
/*
* string_agg - Concatenates values and returns string.
*
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index eea9f30a79..7332f93a25 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -160,6 +160,7 @@ typedef struct _dumpOptions
int no_subscriptions;
int no_synchronized_snapshots;
int no_unlogged_table_data;
+ int no_compression_methods;
int serializable_deferrable;
int disable_triggers;
int outputNoTablespaces;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index eb988d7eb4..46044cb92a 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -387,6 +387,7 @@ main(int argc, char **argv)
{"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1},
{"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1},
{"no-subscriptions", no_argument, &dopt.no_subscriptions, 1},
+ {"no-compression-methods", no_argument, &dopt.no_compression_methods, 1},
{"no-sync", no_argument, NULL, 7},
{"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1},
{"rows-per-insert", required_argument, NULL, 10},
@@ -1047,6 +1048,7 @@ help(const char *progname)
printf(_(" --no-publications do not dump publications\n"));
printf(_(" --no-security-labels do not dump security label assignments\n"));
printf(_(" --no-subscriptions do not dump subscriptions\n"));
+ printf(_(" --no-compression-methods do not dump compression methods\n"));
printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n"));
printf(_(" --no-tablespaces do not dump tablespace assignments\n"));
printf(_(" --no-unlogged-table-data do not dump unlogged table data\n"));
@@ -8617,6 +8619,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
{
DumpOptions *dopt = fout->dopt;
PQExpBuffer q = createPQExpBuffer();
+ bool createWithCompression;
for (int i = 0; i < numTables; i++)
{
@@ -8702,6 +8705,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
appendPQExpBufferStr(q,
"'' AS attidentity,\n");
+ createWithCompression = (fout->remoteVersion >= 140000);
+
+ if (createWithCompression)
+ appendPQExpBuffer(q,
+ "am.amname AS attcmname,\n");
+ else
+ appendPQExpBuffer(q,
+ "NULL AS attcmname,\n");
+
if (fout->remoteVersion >= 110000)
appendPQExpBufferStr(q,
"CASE WHEN a.atthasmissing AND NOT a.attisdropped "
@@ -8720,7 +8732,12 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
/* need left join here to not fail on dropped columns ... */
appendPQExpBuffer(q,
"FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t "
- "ON a.atttypid = t.oid\n"
+ "ON a.atttypid = t.oid\n");
+
+ if (createWithCompression)
+ appendPQExpBuffer(q, "LEFT JOIN pg_catalog.pg_am am "
+ "ON a.attcompression = am.oid\n");
+ appendPQExpBuffer(q,
"WHERE a.attrelid = '%u'::pg_catalog.oid "
"AND a.attnum > 0::pg_catalog.int2\n"
"ORDER BY a.attnum",
@@ -8747,6 +8764,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid));
tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *));
tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *));
+ tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *));
tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *));
@@ -8775,6 +8793,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation")));
tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions")));
tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval")));
+ tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname")));
tbinfo->attrdefs[j] = NULL; /* fix below */
if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't')
hasdefaults = true;
@@ -15832,6 +15851,7 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
{
bool print_default;
bool print_notnull;
+ bool has_non_default_compression;
/*
* Default value --- suppress if to be printed separately.
@@ -15856,6 +15876,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
!dopt->binary_upgrade)
continue;
+ has_non_default_compression = (tbinfo->attcmnames[j] &&
+ (strcmp(tbinfo->attcmnames[j], "pglz") != 0));
+
/* Format properly if not first attr */
if (actual_atts == 0)
appendPQExpBufferStr(q, " (");
@@ -15891,6 +15914,17 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
tbinfo->atttypnames[j]);
}
+ /*
+ * Attribute compression
+ */
+ if (!dopt->no_compression_methods &&
+ tbinfo->attcmnames[j] && strlen(tbinfo->attcmnames[j]) &&
+ (has_non_default_compression || dopt->binary_upgrade))
+ {
+ appendPQExpBuffer(q, " COMPRESSION %s",
+ tbinfo->attcmnames[j]);
+ }
+
if (print_default)
{
if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 0a2213fb06..1789e18f46 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -326,6 +326,7 @@ typedef struct _tableInfo
char *partbound; /* partition bound definition */
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
+ char **attcmnames; /* per-attribute current compression method */
/*
* Stuff computed only for dumpable tables.
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 737e46464a..97791f8dbe 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -2284,9 +2284,9 @@ my %tests = (
regexp => qr/^
\QCREATE TABLE dump_test.test_table (\E\n
\s+\Qcol1 integer NOT NULL,\E\n
- \s+\Qcol2 text,\E\n
- \s+\Qcol3 text,\E\n
- \s+\Qcol4 text,\E\n
+ \s+\Qcol2 text\E\D*,\n
+ \s+\Qcol3 text\E\D*,\n
+ \s+\Qcol4 text\E\D*,\n
\s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n
\Q)\E\n
\QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm,
@@ -2326,7 +2326,7 @@ my %tests = (
regexp => qr/^
\QCREATE TABLE dump_test.test_second_table (\E
\n\s+\Qcol1 integer,\E
- \n\s+\Qcol2 text\E
+ \n\s+\Qcol2 text\E\D*
\n\);
/xm,
like =>
@@ -2441,7 +2441,7 @@ my %tests = (
\n\s+\Qcol1 integer,\E
\n\s+\Qcol2 boolean,\E
\n\s+\Qcol3 boolean,\E
- \n\s+\Qcol4 bit(5),\E
+ \n\s+\Qcol4 bit(5)\E\D*,
\n\s+\Qcol5 double precision\E
\n\);
/xm,
@@ -2459,7 +2459,7 @@ my %tests = (
regexp => qr/^
\QCREATE TABLE dump_test.test_table_identity (\E\n
\s+\Qcol1 integer NOT NULL,\E\n
- \s+\Qcol2 text\E\n
+ \s+\Qcol2 text\E\D*\n
\);
.*
\QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 20af5a92b4..ba464d463e 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -170,10 +170,12 @@ describeAccessMethods(const char *pattern, bool verbose)
" CASE amtype"
" WHEN 'i' THEN '%s'"
" WHEN 't' THEN '%s'"
+ " WHEN 'c' THEN '%s'"
" END AS \"%s\"",
gettext_noop("Name"),
gettext_noop("Index"),
gettext_noop("Table"),
+ gettext_noop("Compression"),
gettext_noop("Type"));
if (verbose)
@@ -1459,7 +1461,7 @@ describeOneTableDetails(const char *schemaname,
bool printTableInitialized = false;
int i;
char *view_def = NULL;
- char *headers[11];
+ char *headers[12];
PQExpBufferData title;
PQExpBufferData tmpbuf;
int cols;
@@ -1475,7 +1477,8 @@ describeOneTableDetails(const char *schemaname,
fdwopts_col = -1,
attstorage_col = -1,
attstattarget_col = -1,
- attdescr_col = -1;
+ attdescr_col = -1,
+ attcompression_col = -1;
int numrows;
struct
{
@@ -1892,6 +1895,20 @@ describeOneTableDetails(const char *schemaname,
appendPQExpBufferStr(&buf, ",\n a.attstorage");
attstorage_col = cols++;
+ /* compresssion info */
+ if (pset.sversion >= 140000 &&
+ (tableinfo.relkind == RELKIND_RELATION ||
+ tableinfo.relkind == RELKIND_PARTITIONED_TABLE ||
+ tableinfo.relkind == RELKIND_MATVIEW))
+ {
+ appendPQExpBufferStr(&buf, ",\n CASE WHEN attcompression = 0 THEN NULL ELSE "
+ " (SELECT am.amname "
+ " FROM pg_catalog.pg_am am "
+ " WHERE am.oid = a.attcompression) "
+ " END AS attcompression");
+ attcompression_col = cols++;
+ }
+
/* stats target, if relevant to relkind */
if (tableinfo.relkind == RELKIND_RELATION ||
tableinfo.relkind == RELKIND_INDEX ||
@@ -2018,6 +2035,8 @@ describeOneTableDetails(const char *schemaname,
headers[cols++] = gettext_noop("FDW options");
if (attstorage_col >= 0)
headers[cols++] = gettext_noop("Storage");
+ if (attcompression_col >= 0)
+ headers[cols++] = gettext_noop("Compression");
if (attstattarget_col >= 0)
headers[cols++] = gettext_noop("Stats target");
if (attdescr_col >= 0)
@@ -2097,6 +2116,11 @@ describeOneTableDetails(const char *schemaname,
false, false);
}
+ /* Column compression */
+ if (attcompression_col >= 0)
+ printTableAddCell(&cont, PQgetvalue(res, i, attcompression_col),
+ false, false);
+
/* Statistics target, if the relkind supports this feature */
if (attstattarget_col >= 0)
printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col),
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
new file mode 100644
index 0000000000..5a8e23d926
--- /dev/null
+++ b/src/include/access/compressamapi.h
@@ -0,0 +1,99 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressamapi.h
+ * API for Postgres compression methods.
+ *
+ * Portions Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ * src/include/access/compressamapi.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef COMPRESSAMAPI_H
+#define COMPRESSAMAPI_H
+
+#include "postgres.h"
+
+#include "catalog/pg_am_d.h"
+#include "nodes/nodes.h"
+
+/*
+ * Built-in compression method-id. The toast compression header will store
+ * this in the first 2 bits of the raw length. These built-in compression
+ * method-id are directly mapped to the built-in compression method oid.
+ */
+typedef enum CompressionId
+{
+ PGLZ_COMPRESSION_ID = 0,
+ LZ4_COMPRESSION_ID = 1
+} CompressionId;
+
+/* Use default compression method if it is not specified. */
+#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID
+#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
+ (storage) != TYPSTORAGE_EXTERNAL)
+/* compression handler routines */
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmdecompress_slice_function)
+ (const struct varlena *value, int32 slicelength);
+
+/*
+ * API struct for a compression AM.
+ *
+ * 'datum_compress' - varlena compression function.
+ * 'datum_decompress' - varlena decompression function.
+ * 'datum_decompress_slice' - varlena slice decompression functions.
+ */
+typedef struct CompressionAmRoutine
+{
+ NodeTag type;
+
+ cmcompress_function datum_compress;
+ cmdecompress_function datum_decompress;
+ cmdecompress_slice_function datum_decompress_slice;
+} CompressionAmRoutine;
+
+extern const CompressionAmRoutine pglz_compress_methods;
+extern const CompressionAmRoutine lz4_compress_methods;
+
+/*
+ * CompressionOidToId - Convert compression Oid to built-in compression id.
+ *
+ * For more details refer comment atop CompressionId in compressamapi.h
+ */
+static inline CompressionId
+CompressionOidToId(Oid cmoid)
+{
+ switch (cmoid)
+ {
+ case PGLZ_COMPRESSION_AM_OID:
+ return PGLZ_COMPRESSION_ID;
+ case LZ4_COMPRESSION_AM_OID:
+ return LZ4_COMPRESSION_ID;
+ default:
+ elog(ERROR, "invalid compression method oid %u", cmoid);
+ }
+}
+
+/*
+ * CompressionIdToOid - Convert built-in compression id to Oid
+ *
+ * For more details refer comment atop CompressionId in compressamapi.h
+ */
+static inline Oid
+CompressionIdToOid(CompressionId cmid)
+{
+ switch (cmid)
+ {
+ case PGLZ_COMPRESSION_ID:
+ return PGLZ_COMPRESSION_AM_OID;
+ case LZ4_COMPRESSION_ID:
+ return LZ4_COMPRESSION_AM_OID;
+ default:
+ elog(ERROR, "invalid compression method id %d", cmid);
+ }
+}
+
+#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h
index 0adf53c77b..6cdc37542d 100644
--- a/src/include/access/detoast.h
+++ b/src/include/access/detoast.h
@@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value);
*/
extern Size toast_datum_size(Datum value);
+/* ----------
+ * toast_get_compression_oid -
+ *
+ * Return the compression method oid from the compressed value
+ * ----------
+ */
+extern Oid toast_get_compression_oid(struct varlena *attr);
+
#endif /* DETOAST_H */
diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h
index a9a6d644bc..dca0bc37f3 100644
--- a/src/include/access/toast_helper.h
+++ b/src/include/access/toast_helper.h
@@ -32,6 +32,7 @@ typedef struct
struct varlena *tai_oldexternal;
int32 tai_size;
uint8 tai_colflags;
+ Oid tai_compression;
} ToastAttrInfo;
/*
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index cedfb890d8..31ff91a09c 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -12,6 +12,7 @@
#ifndef TOAST_INTERNALS_H
#define TOAST_INTERNALS_H
+#include "access/compressamapi.h"
#include "storage/lockdefs.h"
#include "utils/relcache.h"
#include "utils/snapshot.h"
@@ -22,22 +23,22 @@
typedef struct toast_compress_header
{
int32 vl_len_; /* varlena header (do not touch directly!) */
- int32 rawsize;
+ uint32 info; /* 2 bits for compression method and 30 bits
+ * rawsize */
} toast_compress_header;
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
-#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
-#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize)
-#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ)
-#define TOAST_COMPRESS_RAWDATA(ptr) \
- (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ)
-#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \
- (((toast_compress_header *) (ptr))->rawsize = (len))
+#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
+#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
+ do { \
+ Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \
+ ((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
+ } while (0)
-extern Datum toast_compress_datum(Datum value);
+extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative);
diff --git a/src/include/catalog/pg_am.dat b/src/include/catalog/pg_am.dat
index 6082f0e6a8..605684408c 100644
--- a/src/include/catalog/pg_am.dat
+++ b/src/include/catalog/pg_am.dat
@@ -33,5 +33,11 @@
{ oid => '3580', oid_symbol => 'BRIN_AM_OID',
descr => 'block range index (BRIN) access method',
amname => 'brin', amhandler => 'brinhandler', amtype => 'i' },
+{ oid => '4572', oid_symbol => 'PGLZ_COMPRESSION_AM_OID',
+ descr => 'pglz compression access method',
+ amname => 'pglz', amhandler => 'pglzhandler', amtype => 'c' },
+{ oid => '4573', oid_symbol => 'LZ4_COMPRESSION_AM_OID',
+ descr => 'lz4 compression access method',
+ amname => 'lz4', amhandler => 'lz4handler', amtype => 'c' },
]
diff --git a/src/include/catalog/pg_am.h b/src/include/catalog/pg_am.h
index ced86faef8..65079f3821 100644
--- a/src/include/catalog/pg_am.h
+++ b/src/include/catalog/pg_am.h
@@ -59,6 +59,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_am_oid_index, 2652, on pg_am using btree(oid oid_op
*/
#define AMTYPE_INDEX 'i' /* index access method */
#define AMTYPE_TABLE 't' /* table access method */
+#define AMTYPE_COMPRESSION 'c' /* compression access method */
#endif /* EXPOSE_TO_CLIENT_CODE */
diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h
index 3db42abf08..797e78b17c 100644
--- a/src/include/catalog/pg_attribute.h
+++ b/src/include/catalog/pg_attribute.h
@@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75,
/* attribute's collation, if any */
Oid attcollation BKI_LOOKUP_OPT(pg_collation);
+ /*
+ * OID of compression AM. Must be InvalidOid if and only if typstorage is
+ * 'plain' or 'external'.
+ */
+ Oid attcompression BKI_DEFAULT(0);
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* NOTE: The following fields are not present in tuple descriptors. */
@@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75,
* can access fields beyond attcollation except in a real tuple!
*/
#define ATTRIBUTE_FIXED_PART_SIZE \
- (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid))
+ (offsetof(FormData_pg_attribute,attcompression) + sizeof(Oid))
/* ----------------
* Form_pg_attribute corresponds to a pointer to a tuple with
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 1487710d59..4b3d34ae1a 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -941,6 +941,15 @@
prorettype => 'void', proargtypes => 'regclass int8',
prosrc => 'brin_desummarize_range' },
+{ oid => '6015', descr => 'pglz compression access method handler',
+ proname => 'pglzhandler', provolatile => 'v',
+ prorettype => 'compression_am_handler', proargtypes => 'internal',
+ prosrc => 'pglzhandler' },
+{ oid => '6016', descr => 'lz4 compression access method handler',
+ proname => 'lz4handler', provolatile => 'v',
+ prorettype => 'compression_am_handler', proargtypes => 'internal',
+ prosrc => 'lz4handler' },
+
{ oid => '338', descr => 'validate an operator class',
proname => 'amvalidate', provolatile => 'v', prorettype => 'bool',
proargtypes => 'oid', prosrc => 'amvalidate' },
@@ -7095,6 +7104,10 @@
descr => 'bytes required to store the value, perhaps with compression',
proname => 'pg_column_size', provolatile => 's', prorettype => 'int4',
proargtypes => 'any', prosrc => 'pg_column_size' },
+{ oid => '2121',
+ descr => 'compression method for the compressed datum',
+ proname => 'pg_column_compression', provolatile => 's', prorettype => 'text',
+ proargtypes => 'any', prosrc => 'pg_column_compression' },
{ oid => '2322',
descr => 'total disk space usage for the specified tablespace',
proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8',
@@ -7265,6 +7278,13 @@
{ oid => '268', descr => 'I/O',
proname => 'table_am_handler_out', prorettype => 'cstring',
proargtypes => 'table_am_handler', prosrc => 'table_am_handler_out' },
+{ oid => '560', descr => 'I/O',
+ proname => 'compression_am_handler_in', proisstrict => 'f',
+ prorettype => 'compression_am_handler', proargtypes => 'cstring',
+ prosrc => 'compression_am_handler_in' },
+{ oid => '561', descr => 'I/O',
+ proname => 'compression_am_handler_out', prorettype => 'cstring',
+ proargtypes => 'compression_am_handler', prosrc => 'compression_am_handler_out' },
{ oid => '5086', descr => 'I/O',
proname => 'anycompatible_in', prorettype => 'anycompatible',
proargtypes => 'cstring', prosrc => 'anycompatible_in' },
diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat
index 8959c2f53b..306aabf6c1 100644
--- a/src/include/catalog/pg_type.dat
+++ b/src/include/catalog/pg_type.dat
@@ -626,6 +626,11 @@
typcategory => 'P', typinput => 'index_am_handler_in',
typoutput => 'index_am_handler_out', typreceive => '-', typsend => '-',
typalign => 'i' },
+ { oid => '5559',
+ typname => 'compression_am_handler', typlen => '4', typbyval => 't', typtype => 'p',
+ typcategory => 'P', typinput => 'compression_am_handler_in',
+ typoutput => 'compression_am_handler_out', typreceive => '-', typsend => '-',
+ typalign => 'i' },
{ oid => '3310',
descr => 'pseudo-type for the result of a tablesample method function',
typname => 'tsm_handler', typlen => '4', typbyval => 't', typtype => 'p',
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index 1a79540c94..e5aea8a240 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -139,6 +139,7 @@ extern Datum transformGenericOptions(Oid catalogId,
extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt);
extern Oid get_index_am_oid(const char *amname, bool missing_ok);
extern Oid get_table_am_oid(const char *amname, bool missing_ok);
+extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index 071e363d54..6495162a33 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd);
extern void CheckSubscriptionRelkind(char relkind, const char *nspname,
const char *relname);
-
+extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
+ TupleTableSlot **outslot,
+ TupleDesc targetTupDesc);
#endif /* EXECUTOR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 943931f65d..ac3884b3a8 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1185,6 +1185,12 @@ typedef struct ModifyTableState
*/
TupleTableSlot *mt_root_tuple_slot;
+ /*
+ * Slot for storing the modified tuple, incase the target attribute's
+ * compression method doesn't match with the source table.
+ */
+ TupleTableSlot *mt_decompress_tuple_slot;
+
/* Tuple-routing support info */
struct PartitionTupleRouting *mt_partition_tuple_routing;
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 40ae489c23..20d6f96f62 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -512,6 +512,7 @@ typedef enum NodeTag
T_IndexAmRoutine, /* in access/amapi.h */
T_TableAmRoutine, /* in access/tableam.h */
T_TsmRoutine, /* in access/tsmapi.h */
+ T_CompressionAmRoutine, /* in access/compressamapi.h */
T_ForeignKeyCacheInfo, /* in utils/rel.h */
T_CallContext, /* in nodes/parsenodes.h */
T_SupportRequestSimplify, /* in nodes/supportnodes.h */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 236832a2ca..19d2ba26bf 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -646,6 +646,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
+ char *compression; /* compression method for column */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
@@ -685,6 +686,7 @@ typedef enum TableLikeOption
CREATE_TABLE_LIKE_INDEXES = 1 << 5,
CREATE_TABLE_LIKE_STATISTICS = 1 << 6,
CREATE_TABLE_LIKE_STORAGE = 1 << 7,
+ CREATE_TABLE_LIKE_COMPRESSION = 1 << 8,
CREATE_TABLE_LIKE_ALL = PG_INT32_MAX
} TableLikeOption;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..ca1f950cbe 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 55cab4d2bf..53d378b67d 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -346,6 +346,9 @@
/* Define to 1 if you have the `z' library (-lz). */
#undef HAVE_LIBZ
+/* Define to 1 if you have the `lz4' library (-llz4). */
+#undef HAVE_LIBLZ4
+
/* Define to 1 if you have the `link' function. */
#undef HAVE_LINK
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 2ed572004d..667927fd7c 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -145,7 +145,8 @@ typedef union
struct /* Compressed-in-line format */
{
uint32 va_header;
- uint32 va_rawsize; /* Original data size (excludes header) */
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
} varattrib_4b;
@@ -274,14 +275,21 @@ typedef struct
(VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT)
#define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data)
+#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data)
#define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data)
#define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data)
#define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data)
#define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data)
+#define VARLENA_RAWSIZE_BITS 30
+#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1)
+
+/* va_info in va_compress contains raw size of datum and optional flags */
#define VARRAWSIZE_4B_C(PTR) \
- (((varattrib_4b *) (PTR))->va_compressed.va_rawsize)
+ (((varattrib_4b *) (PTR))->va_compressed.va_info & VARLENA_RAWSIZE_MASK)
+#define VARFLAGS_4B_C(PTR) \
+ (((varattrib_4b *) (PTR))->va_compressed.va_info >> VARLENA_RAWSIZE_BITS)
/* Externally visible macros */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
new file mode 100644
index 0000000000..167878e78b
--- /dev/null
+++ b/src/test/regress/expected/compression.out
@@ -0,0 +1,201 @@
+\set HIDE_COMPRESSAM off
+-- test creating table with compression method
+CREATE TABLE cmdata(f1 text COMPRESSION pglz);
+CREATE INDEX idx ON cmdata(f1);
+INSERT INTO cmdata VALUES(repeat('1234567890',1000));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+------+-----------+----------+---------+----------+--------------+-------------
+ f1 | text | | | | extended | |
+Indexes:
+ "idx" btree (f1)
+
+CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4);
+INSERT INTO cmdata1 VALUES(repeat('1234567890',1004));
+\d+ cmdata1
+ Table "public.cmdata1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+------+-----------+----------+---------+----------+--------------+-------------
+ f1 | text | | | | extended | |
+
+-- try setting compression for incompressible data type
+CREATE TABLE cmdata2 (f1 int COMPRESSION pglz);
+ERROR: column data type integer does not support compression
+-- verify stored compression method
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+SELECT pg_column_compression(f1) FROM cmdata1;
+ pg_column_compression
+-----------------------
+ lz4
+(1 row)
+
+-- decompress data slice
+SELECT SUBSTR(f1, 200, 5) FROM cmdata;
+ substr
+--------
+ 01234
+(1 row)
+
+SELECT SUBSTR(f1, 2000, 50) FROM cmdata1;
+ substr
+----------------------------------------------------
+ 01234567890123456789012345678901234567890123456789
+(1 row)
+
+-- copy with table creation
+SELECT * INTO cmmove1 FROM cmdata;
+SELECT pg_column_compression(f1) FROM cmmove1;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+-- update using datum from different table
+CREATE TABLE cmmove2(f1 text COMPRESSION pglz);
+INSERT INTO cmmove2 VALUES (repeat('1234567890',1004));
+SELECT pg_column_compression(f1) FROM cmmove2;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata;
+SELECT pg_column_compression(f1) FROM cmmove2;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1;
+SELECT pg_column_compression(f1) FROM cmmove2;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+-- copy to existing table
+CREATE TABLE cmmove3(f1 text COMPRESSION pglz);
+INSERT INTO cmmove3 SELECT * FROM cmdata;
+INSERT INTO cmmove3 SELECT * FROM cmdata1;
+SELECT pg_column_compression(f1) FROM cmmove2;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+-- test external compressed data
+CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS
+'select array_agg(md5(g::text))::text from generate_series(1, 256) g';
+CREATE TABLE cmdata2 (f1 text COMPRESSION pglz);
+INSERT INTO cmdata2 select large_val() || repeat('a', 4000);
+SELECT pg_column_compression(f1) FROM cmdata2;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+INSERT INTO cmdata1 SELECT * FROM cmdata2;
+SELECT pg_column_compression(f1) FROM cmdata1;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+(2 rows)
+
+DROP TABLE cmdata2;
+-- test LIKE INCLUDING COMPRESSION
+CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION);
+\d+ cmdata2
+ Table "public.cmdata2"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+------+-----------+----------+---------+----------+--------------+-------------
+ f1 | text | | | | extended | |
+
+-- test compression with materialized view
+CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1;
+\d+ mv
+ Materialized view "public.mv"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+------+-----------+----------+---------+----------+--------------+-------------
+ x | text | | | | extended | |
+View definition:
+ SELECT cmdata1.f1 AS x
+ FROM cmdata1;
+
+SELECT pg_column_compression(f1) FROM cmdata1;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+(2 rows)
+
+SELECT pg_column_compression(x) FROM mv;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+-- test compression with partition
+CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1);
+CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0);
+CREATE TABLE cmpart2(f1 text COMPRESSION pglz);
+ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1);
+INSERT INTO cmpart VALUES (repeat('123456789',1004));
+INSERT INTO cmpart VALUES (repeat('123456789',4004));
+SELECT pg_column_compression(f1) FROM cmpart;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+-- test compression with inheritence, error
+CREATE TABLE cminh() INHERITS(cmdata, cmdata1);
+NOTICE: merging multiple inherited definitions of column "f1"
+ERROR: column "f1" has a compression method conflict
+DETAIL: pglz versus lz4
+CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata);
+NOTICE: merging column "f1" with inherited definition
+ERROR: column "f1" has a compression method conflict
+DETAIL: pglz versus lz4
+-- check data is ok
+SELECT length(f1) FROM cmdata;
+ length
+--------
+ 10000
+(1 row)
+
+SELECT length(f1) FROM cmdata1;
+ length
+--------
+ 10040
+ 12449
+(2 rows)
+
+SELECT length(f1) FROM cmmove1;
+ length
+--------
+ 10000
+(1 row)
+
+SELECT length(f1) FROM cmmove2;
+ length
+--------
+ 10040
+(1 row)
+
+SELECT length(f1) FROM cmmove3;
+ length
+--------
+ 10000
+ 10040
+(2 rows)
+
+\set HIDE_COMPRESSAM on
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
new file mode 100644
index 0000000000..329e7881b0
--- /dev/null
+++ b/src/test/regress/expected/compression_1.out
@@ -0,0 +1,189 @@
+\set HIDE_COMPRESSAM off
+-- test creating table with compression method
+CREATE TABLE cmdata(f1 text COMPRESSION pglz);
+CREATE INDEX idx ON cmdata(f1);
+INSERT INTO cmdata VALUES(repeat('1234567890',1000));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+------+-----------+----------+---------+----------+--------------+-------------
+ f1 | text | | | | extended | |
+Indexes:
+ "idx" btree (f1)
+
+CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4);
+ERROR: not built with lz4 support
+INSERT INTO cmdata1 VALUES(repeat('1234567890',1004));
+ERROR: relation "cmdata1" does not exist
+LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004));
+ ^
+\d+ cmdata1
+-- try setting compression for incompressible data type
+CREATE TABLE cmdata2 (f1 int COMPRESSION pglz);
+ERROR: column data type integer does not support compression
+-- verify stored compression method
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+SELECT pg_column_compression(f1) FROM cmdata1;
+ERROR: relation "cmdata1" does not exist
+LINE 1: SELECT pg_column_compression(f1) FROM cmdata1;
+ ^
+-- decompress data slice
+SELECT SUBSTR(f1, 200, 5) FROM cmdata;
+ substr
+--------
+ 01234
+(1 row)
+
+SELECT SUBSTR(f1, 2000, 50) FROM cmdata1;
+ERROR: relation "cmdata1" does not exist
+LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1;
+ ^
+-- copy with table creation
+SELECT * INTO cmmove1 FROM cmdata;
+SELECT pg_column_compression(f1) FROM cmmove1;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+-- update using datum from different table
+CREATE TABLE cmmove2(f1 text COMPRESSION pglz);
+INSERT INTO cmmove2 VALUES (repeat('1234567890',1004));
+SELECT pg_column_compression(f1) FROM cmmove2;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata;
+SELECT pg_column_compression(f1) FROM cmmove2;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1;
+ERROR: relation "cmdata1" does not exist
+LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1;
+ ^
+SELECT pg_column_compression(f1) FROM cmmove2;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+-- copy to existing table
+CREATE TABLE cmmove3(f1 text COMPRESSION pglz);
+INSERT INTO cmmove3 SELECT * FROM cmdata;
+INSERT INTO cmmove3 SELECT * FROM cmdata1;
+ERROR: relation "cmdata1" does not exist
+LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1;
+ ^
+SELECT pg_column_compression(f1) FROM cmmove2;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+-- test external compressed data
+CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS
+'select array_agg(md5(g::text))::text from generate_series(1, 256) g';
+CREATE TABLE cmdata2 (f1 text COMPRESSION pglz);
+INSERT INTO cmdata2 select large_val() || repeat('a', 4000);
+SELECT pg_column_compression(f1) FROM cmdata2;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+INSERT INTO cmdata1 SELECT * FROM cmdata2;
+ERROR: relation "cmdata1" does not exist
+LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2;
+ ^
+SELECT pg_column_compression(f1) FROM cmdata1;
+ERROR: relation "cmdata1" does not exist
+LINE 1: SELECT pg_column_compression(f1) FROM cmdata1;
+ ^
+DROP TABLE cmdata2;
+-- test LIKE INCLUDING COMPRESSION
+CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION);
+ERROR: relation "cmdata1" does not exist
+LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION);
+ ^
+\d+ cmdata2
+-- test compression with materialized view
+CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1;
+ERROR: relation "cmdata1" does not exist
+LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1;
+ ^
+\d+ mv
+SELECT pg_column_compression(f1) FROM cmdata1;
+ERROR: relation "cmdata1" does not exist
+LINE 1: SELECT pg_column_compression(f1) FROM cmdata1;
+ ^
+SELECT pg_column_compression(x) FROM mv;
+ERROR: relation "mv" does not exist
+LINE 1: SELECT pg_column_compression(x) FROM mv;
+ ^
+-- test compression with partition
+CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1);
+ERROR: not built with lz4 support
+CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0);
+ERROR: relation "cmpart" does not exist
+CREATE TABLE cmpart2(f1 text COMPRESSION pglz);
+ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1);
+ERROR: relation "cmpart" does not exist
+INSERT INTO cmpart VALUES (repeat('123456789',1004));
+ERROR: relation "cmpart" does not exist
+LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004));
+ ^
+INSERT INTO cmpart VALUES (repeat('123456789',4004));
+ERROR: relation "cmpart" does not exist
+LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004));
+ ^
+SELECT pg_column_compression(f1) FROM cmpart;
+ERROR: relation "cmpart" does not exist
+LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
+ ^
+-- test compression with inheritence, error
+CREATE TABLE cminh() INHERITS(cmdata, cmdata1);
+ERROR: relation "cmdata1" does not exist
+CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata);
+NOTICE: merging column "f1" with inherited definition
+ERROR: column "f1" has a compression method conflict
+DETAIL: pglz versus lz4
+-- check data is ok
+SELECT length(f1) FROM cmdata;
+ length
+--------
+ 10000
+(1 row)
+
+SELECT length(f1) FROM cmdata1;
+ERROR: relation "cmdata1" does not exist
+LINE 1: SELECT length(f1) FROM cmdata1;
+ ^
+SELECT length(f1) FROM cmmove1;
+ length
+--------
+ 10000
+(1 row)
+
+SELECT length(f1) FROM cmmove2;
+ length
+--------
+ 10000
+(1 row)
+
+SELECT length(f1) FROM cmmove3;
+ length
+--------
+ 10000
+(1 row)
+
+\set HIDE_COMPRESSAM on
diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out
index 7204fdb0b4..14f3fa3fc8 100644
--- a/src/test/regress/expected/psql.out
+++ b/src/test/regress/expected/psql.out
@@ -4898,8 +4898,8 @@ Indexes:
-- check printing info about access methods
\dA
List of access methods
- Name | Type
---------+-------
+ Name | Type
+--------+-------------
brin | Index
btree | Index
gin | Index
@@ -4907,13 +4907,15 @@ List of access methods
hash | Index
heap | Table
heap2 | Table
+ lz4 | Compression
+ pglz | Compression
spgist | Index
-(8 rows)
+(10 rows)
\dA *
List of access methods
- Name | Type
---------+-------
+ Name | Type
+--------+-------------
brin | Index
btree | Index
gin | Index
@@ -4921,8 +4923,10 @@ List of access methods
hash | Index
heap | Table
heap2 | Table
+ lz4 | Compression
+ pglz | Compression
spgist | Index
-(8 rows)
+(10 rows)
\dA h*
List of access methods
@@ -4947,32 +4951,36 @@ List of access methods
\dA: extra argument "bar" ignored
\dA+
- List of access methods
- Name | Type | Handler | Description
---------+-------+----------------------+----------------------------------------
- brin | Index | brinhandler | block range index (BRIN) access method
- btree | Index | bthandler | b-tree index access method
- gin | Index | ginhandler | GIN index access method
- gist | Index | gisthandler | GiST index access method
- hash | Index | hashhandler | hash index access method
- heap | Table | heap_tableam_handler | heap table access method
- heap2 | Table | heap_tableam_handler |
- spgist | Index | spghandler | SP-GiST index access method
-(8 rows)
+ List of access methods
+ Name | Type | Handler | Description
+--------+-------------+----------------------+----------------------------------------
+ brin | Index | brinhandler | block range index (BRIN) access method
+ btree | Index | bthandler | b-tree index access method
+ gin | Index | ginhandler | GIN index access method
+ gist | Index | gisthandler | GiST index access method
+ hash | Index | hashhandler | hash index access method
+ heap | Table | heap_tableam_handler | heap table access method
+ heap2 | Table | heap_tableam_handler |
+ lz4 | Compression | lz4handler | lz4 compression access method
+ pglz | Compression | pglzhandler | pglz compression access method
+ spgist | Index | spghandler | SP-GiST index access method
+(10 rows)
\dA+ *
- List of access methods
- Name | Type | Handler | Description
---------+-------+----------------------+----------------------------------------
- brin | Index | brinhandler | block range index (BRIN) access method
- btree | Index | bthandler | b-tree index access method
- gin | Index | ginhandler | GIN index access method
- gist | Index | gisthandler | GiST index access method
- hash | Index | hashhandler | hash index access method
- heap | Table | heap_tableam_handler | heap table access method
- heap2 | Table | heap_tableam_handler |
- spgist | Index | spghandler | SP-GiST index access method
-(8 rows)
+ List of access methods
+ Name | Type | Handler | Description
+--------+-------------+----------------------+----------------------------------------
+ brin | Index | brinhandler | block range index (BRIN) access method
+ btree | Index | bthandler | b-tree index access method
+ gin | Index | ginhandler | GIN index access method
+ gist | Index | gisthandler | GiST index access method
+ hash | Index | hashhandler | hash index access method
+ heap | Table | heap_tableam_handler | heap table access method
+ heap2 | Table | heap_tableam_handler |
+ lz4 | Compression | lz4handler | lz4 compression access method
+ pglz | Compression | pglzhandler | pglz compression access method
+ spgist | Index | spghandler | SP-GiST index access method
+(10 rows)
\dA+ h*
List of access methods
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 12bb67e491..e9c0fc9760 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -114,7 +114,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr
# ----------
# Another group of parallel tests
# ----------
-test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain
+test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression
# event triggers cannot run concurrently with any test that runs DDL
# oidjoins is read-only, though, and should run late for best coverage
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index 59b416fd80..4d5577bae3 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -199,6 +199,7 @@ test: partition_aggregate
test: partition_info
test: tuplesort
test: explain
+test: compression
test: event_trigger
test: oidjoins
test: fast_default
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
new file mode 100644
index 0000000000..450416ecb4
--- /dev/null
+++ b/src/test/regress/sql/compression.sql
@@ -0,0 +1,85 @@
+\set HIDE_COMPRESSAM off
+
+-- test creating table with compression method
+CREATE TABLE cmdata(f1 text COMPRESSION pglz);
+CREATE INDEX idx ON cmdata(f1);
+INSERT INTO cmdata VALUES(repeat('1234567890',1000));
+\d+ cmdata
+
+CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4);
+INSERT INTO cmdata1 VALUES(repeat('1234567890',1004));
+\d+ cmdata1
+
+-- try setting compression for incompressible data type
+CREATE TABLE cmdata2 (f1 int COMPRESSION pglz);
+
+-- verify stored compression method
+SELECT pg_column_compression(f1) FROM cmdata;
+SELECT pg_column_compression(f1) FROM cmdata1;
+
+-- decompress data slice
+SELECT SUBSTR(f1, 200, 5) FROM cmdata;
+SELECT SUBSTR(f1, 2000, 50) FROM cmdata1;
+
+-- copy with table creation
+SELECT * INTO cmmove1 FROM cmdata;
+SELECT pg_column_compression(f1) FROM cmmove1;
+
+-- update using datum from different table
+CREATE TABLE cmmove2(f1 text COMPRESSION pglz);
+INSERT INTO cmmove2 VALUES (repeat('1234567890',1004));
+SELECT pg_column_compression(f1) FROM cmmove2;
+
+UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata;
+SELECT pg_column_compression(f1) FROM cmmove2;
+UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1;
+SELECT pg_column_compression(f1) FROM cmmove2;
+
+-- copy to existing table
+CREATE TABLE cmmove3(f1 text COMPRESSION pglz);
+INSERT INTO cmmove3 SELECT * FROM cmdata;
+INSERT INTO cmmove3 SELECT * FROM cmdata1;
+SELECT pg_column_compression(f1) FROM cmmove2;
+
+-- test external compressed data
+CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS
+'select array_agg(md5(g::text))::text from generate_series(1, 256) g';
+CREATE TABLE cmdata2 (f1 text COMPRESSION pglz);
+INSERT INTO cmdata2 select large_val() || repeat('a', 4000);
+SELECT pg_column_compression(f1) FROM cmdata2;
+INSERT INTO cmdata1 SELECT * FROM cmdata2;
+SELECT pg_column_compression(f1) FROM cmdata1;
+DROP TABLE cmdata2;
+
+-- test LIKE INCLUDING COMPRESSION
+CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION);
+\d+ cmdata2
+
+-- test compression with materialized view
+CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1;
+\d+ mv
+SELECT pg_column_compression(f1) FROM cmdata1;
+SELECT pg_column_compression(x) FROM mv;
+
+-- test compression with partition
+CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1);
+CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0);
+CREATE TABLE cmpart2(f1 text COMPRESSION pglz);
+
+ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1);
+INSERT INTO cmpart VALUES (repeat('123456789',1004));
+INSERT INTO cmpart VALUES (repeat('123456789',4004));
+SELECT pg_column_compression(f1) FROM cmpart;
+
+-- test compression with inheritence, error
+CREATE TABLE cminh() INHERITS(cmdata, cmdata1);
+CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata);
+
+-- check data is ok
+SELECT length(f1) FROM cmdata;
+SELECT length(f1) FROM cmdata1;
+SELECT length(f1) FROM cmmove1;
+SELECT length(f1) FROM cmmove2;
+SELECT length(f1) FROM cmmove3;
+
+\set HIDE_COMPRESSAM on
diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm
index 2aa062b2c9..c64b369047 100644
--- a/src/tools/msvc/Solution.pm
+++ b/src/tools/msvc/Solution.pm
@@ -307,6 +307,7 @@ sub GenerateFiles
HAVE_LIBXML2 => undef,
HAVE_LIBXSLT => undef,
HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef,
+ HAVE_LIBLZ4 => undef,
HAVE_LINK => undef,
HAVE_LOCALE_T => 1,
HAVE_LONG_INT_64 => undef,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index bab4f3adb3..46cac11d39 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -395,6 +395,7 @@ CompositeIOData
CompositeTypeStmt
CompoundAffixFlag
CompressionAlgorithm
+CompressionAmRoutine
CompressorState
ComputeXidHorizonsResult
ConditionVariable
--
2.17.0
[text/x-diff] 0003-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch (4.3K, ../20210218211314.GC14772@telsasoft.com/4-0003-psql-Add-HIDE_COMPRESSAM-for-regress-testing.patch)
download | inline diff:
From 61b674c345ab155e3edcedebd77f7380f22aacca Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sat, 13 Feb 2021 20:48:51 -0600
Subject: [PATCH 03/12] psql: Add HIDE_COMPRESSAM for regress testing
This avoids churn in regression output, and allows testing with alternate
compression AMs.
---
doc/src/sgml/ref/psql-ref.sgml | 11 +++++++++++
src/bin/psql/describe.c | 1 +
src/bin/psql/help.c | 2 ++
src/bin/psql/settings.h | 1 +
src/bin/psql/startup.c | 9 +++++++++
src/test/regress/pg_regress_main.c | 4 ++--
6 files changed, 26 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml
index 13c1edfa4d..66dcb1b33b 100644
--- a/doc/src/sgml/ref/psql-ref.sgml
+++ b/doc/src/sgml/ref/psql-ref.sgml
@@ -3863,6 +3863,17 @@ bar
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><varname>HIDE_COMPRESSAM</varname></term>
+ <listitem>
+ <para>
+ If this variable is set to <literal>true</literal>, column's
+ compression access method details are not displayed. This is mainly
+ useful for regression tests.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><varname>HIDE_TABLEAM</varname></term>
<listitem>
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index ba464d463e..b835b0cf76 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -1897,6 +1897,7 @@ describeOneTableDetails(const char *schemaname,
/* compresssion info */
if (pset.sversion >= 140000 &&
+ !pset.hide_compressam &&
(tableinfo.relkind == RELKIND_RELATION ||
tableinfo.relkind == RELKIND_PARTITIONED_TABLE ||
tableinfo.relkind == RELKIND_MATVIEW))
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index e44120bf76..a818ee5503 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -375,6 +375,8 @@ helpVariables(unsigned short int pager)
" true if last query failed, else false\n"));
fprintf(output, _(" FETCH_COUNT\n"
" the number of result rows to fetch and display at a time (0 = unlimited)\n"));
+ fprintf(output, _(" HIDE_COMPRESSAM\n"
+ " if set, compression access methods are not displayed\n"));
fprintf(output, _(" HIDE_TABLEAM\n"
" if set, table access methods are not displayed\n"));
fprintf(output, _(" HISTCONTROL\n"
diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h
index d65990059d..9755e8eac6 100644
--- a/src/bin/psql/settings.h
+++ b/src/bin/psql/settings.h
@@ -134,6 +134,7 @@ typedef struct _psqlSettings
bool quiet;
bool singleline;
bool singlestep;
+ bool hide_compressam;
bool hide_tableam;
int fetch_count;
int histsize;
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index 780479c8d7..554b64367d 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -1159,6 +1159,12 @@ show_context_hook(const char *newval)
return true;
}
+static bool
+hide_compressam_hook(const char *newval)
+{
+ return ParseVariableBool(newval, "HIDE_COMPRESSAM", &pset.hide_compressam);
+}
+
static bool
hide_tableam_hook(const char *newval)
{
@@ -1227,6 +1233,9 @@ EstablishVariableSpace(void)
SetVariableHooks(pset.vars, "SHOW_CONTEXT",
show_context_substitute_hook,
show_context_hook);
+ SetVariableHooks(pset.vars, "HIDE_COMPRESSAM",
+ bool_substitute_hook,
+ hide_compressam_hook);
SetVariableHooks(pset.vars, "HIDE_TABLEAM",
bool_substitute_hook,
hide_tableam_hook);
diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c
index 8dc4941c24..07fd3f6a4d 100644
--- a/src/test/regress/pg_regress_main.c
+++ b/src/test/regress/pg_regress_main.c
@@ -78,11 +78,11 @@ psql_start_test(const char *testname,
* against different AMs without unnecessary differences.
*/
offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset,
- "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1",
+ "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1",
bindir ? bindir : "",
bindir ? "/" : "",
dblist->str,
- "HIDE_TABLEAM=\"on\"",
+ "-v HIDE_TABLEAM=on -v HIDE_COMPRESSAM=on",
infile,
outfile);
if (offset >= sizeof(psql_cmd))
--
2.17.0
[text/x-diff] 0004-Add-default_toast_compression-GUC.patch (10.4K, ../20210218211314.GC14772@telsasoft.com/5-0004-Add-default_toast_compression-GUC.patch)
download | inline diff:
From 570692f03eea1c28fa2b097533aa6a0c5c251237 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 29 Jan 2021 21:37:46 -0600
Subject: [PATCH 04/12] Add default_toast_compression GUC
---
src/backend/access/common/tupdesc.c | 2 +-
src/backend/bootstrap/bootstrap.c | 3 +-
src/backend/commands/amcmds.c | 143 +++++++++++++++++++++++++++-
src/backend/commands/tablecmds.c | 4 +-
src/backend/utils/init/postinit.c | 4 +
src/backend/utils/misc/guc.c | 12 +++
src/include/access/amapi.h | 2 +
src/include/access/compressamapi.h | 12 ++-
8 files changed, 176 insertions(+), 6 deletions(-)
diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c
index ca26fab487..7afaea000b 100644
--- a/src/backend/access/common/tupdesc.c
+++ b/src/backend/access/common/tupdesc.c
@@ -668,7 +668,7 @@ TupleDescInitEntry(TupleDesc desc,
att->attcollation = typeForm->typcollation;
if (IsStorageCompressible(typeForm->typstorage))
- att->attcompression = DefaultCompressionOid;
+ att->attcompression = GetDefaultToastCompression();
else
att->attcompression = InvalidOid;
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 9b451eaa71..ec3376cf8a 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -732,8 +732,9 @@ DefineAttr(char *name, char *type, int attnum, int nullness)
attrtypes[attnum]->attcacheoff = -1;
attrtypes[attnum]->atttypmod = -1;
attrtypes[attnum]->attislocal = true;
+
if (IsStorageCompressible(attrtypes[attnum]->attstorage))
- attrtypes[attnum]->attcompression = DefaultCompressionOid;
+ attrtypes[attnum]->attcompression = GetDefaultToastCompression();
else
attrtypes[attnum]->attcompression = InvalidOid;
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 3ad4a61739..1682afd2a4 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -13,8 +13,11 @@
*/
#include "postgres.h"
+#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "access/xact.h"
#include "catalog/catalog.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
@@ -27,13 +30,20 @@
#include "parser/parse_func.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
+#include "utils/guc.h"
+#include "utils/inval.h"
#include "utils/rel.h"
#include "utils/syscache.h"
-
static Oid lookup_am_handler_func(List *handler_name, char amtype);
static const char *get_am_type_string(char amtype);
+/* Compile-time default */
+char *default_toast_compression = DEFAULT_TOAST_COMPRESSION; // maybe needs pg_dump support ?
+/* Invalid means need to lookup the text value in the catalog */
+static Oid default_toast_compression_oid = InvalidOid;
+
+static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue);
/*
* CreateAccessMethod
@@ -277,3 +287,134 @@ lookup_am_handler_func(List *handler_name, char amtype)
return handlerOid;
}
+
+/* check_hook: validate new default_toast_compression */
+bool
+check_default_toast_compression(char **newval, void **extra, GucSource source)
+{
+ if (**newval == '\0')
+ {
+ GUC_check_errdetail("%s cannot be empty.",
+ "default_toast_compression");
+ return false;
+ }
+
+ if (strlen(*newval) >= NAMEDATALEN)
+ {
+ GUC_check_errdetail("%s is too long (maximum %d characters).",
+ "default_toast_compression", NAMEDATALEN - 1);
+ return false;
+ }
+
+ /*
+ * If we aren't inside a transaction, or not connected to a database, we
+ * cannot do the catalog access necessary to verify the method. Must
+ * accept the value on faith.
+ */
+ if (IsTransactionState() && MyDatabaseId != InvalidOid)
+ {
+ if (!OidIsValid(get_compression_am_oid(*newval, true)))
+ {
+ /*
+ * When source == PGC_S_TEST, don't throw a hard error for a
+ * nonexistent table access method, only a NOTICE. See comments in
+ * guc.h.
+ */
+ if (source == PGC_S_TEST)
+ {
+ ereport(NOTICE,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("compression access method \"%s\" does not exist",
+ *newval)));
+ }
+ else
+ {
+ GUC_check_errdetail("Compression access method \"%s\" does not exist.",
+ *newval);
+ return false;
+ }
+ }
+ }
+
+ return true;
+}
+
+/*
+ * assign_default_toast_compression: GUC assign_hook for default_toast_compression
+ */
+void
+assign_default_toast_compression(const char *newval, void *extra)
+{
+ /*
+ * Invalidate setting, forcing it to be looked up as needed.
+ * This avoids trying to do database access during GUC initialization,
+ * or outside a transaction.
+ */
+
+ default_toast_compression = NULL;
+fprintf(stderr, "set to null\n");
+}
+
+
+/*
+ * GetDefaultToastCompression -- get the OID of the current toast compression
+ *
+ * This exists to hide and optimize the use of the default_toast_compression
+ * GUC variable.
+ */
+Oid
+GetDefaultToastCompression(void)
+{
+ /* Avoid catalog access during bootstrap, and for default compression */
+ if (strcmp(default_toast_compression, DEFAULT_TOAST_COMPRESSION) == 0)
+ return PGLZ_COMPRESSION_AM_OID;
+
+ /* Cannot call get_compression_am_oid this early */
+ // if (IsBootstrapProcessingMode())
+ // return PGLZ_COMPRESSION_AM_OID;
+ Assert(!IsBootstrapProcessingMode());
+
+ /*
+ * If cached value isn't valid, look up the current default value, caching
+ * the result
+ */
+ if (!OidIsValid(default_toast_compression_oid))
+ default_toast_compression_oid =
+ get_am_type_oid(default_toast_compression, AMTYPE_COMPRESSION,
+ false);
+
+ return default_toast_compression_oid;
+}
+
+/*
+ * InitializeAccessMethods: initialize module during InitPostgres.
+ *
+ * This is called after we are up enough to be able to do catalog lookups.
+ */
+void
+InitializeAccessMethods(void)
+{
+ if (IsBootstrapProcessingMode())
+ return;
+
+ /*
+ * In normal mode, arrange for a callback on any syscache invalidation
+ * of pg_am rows.
+ */
+ CacheRegisterSyscacheCallback(AMOID,
+ AccessMethodCallback,
+ (Datum) 0);
+ /* Force cached default access method to be recomputed on next use */
+ // default_toast_compression_oid = InvalidOid;
+}
+
+/*
+ * AccessMethodCallback
+ * Syscache inval callback function
+ */
+static void
+AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue)
+{
+ /* Force look up of compression oid on next use */
+ default_toast_compression_oid = false;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 811dbc3741..7adeaedd0e 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -11925,7 +11925,7 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
if (!IsStorageCompressible(tform->typstorage))
attTup->attcompression = InvalidOid;
else if (!OidIsValid(attTup->attcompression))
- attTup->attcompression = DefaultCompressionOid;
+ attTup->attcompression = GetDefaultToastCompression();
}
else
attTup->attcompression = InvalidOid;
@@ -17762,7 +17762,7 @@ GetAttributeCompression(Form_pg_attribute att, char *compression)
/* fallback to default compression if it's not specified */
if (compression == NULL)
- return DefaultCompressionOid;
+ return GetDefaultToastCompression();
amoid = get_compression_am_oid(compression, false);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index e5965bc517..6a02bbd377 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -25,6 +25,7 @@
#include "access/session.h"
#include "access/sysattr.h"
#include "access/tableam.h"
+#include "access/amapi.h"
#include "access/xact.h"
#include "access/xlog.h"
#include "catalog/catalog.h"
@@ -1060,6 +1061,9 @@ InitPostgres(const char *in_dbname, Oid dboid, const char *username,
/* set default namespace search path */
InitializeSearchPath();
+ /* set callback for changes to pg_am */
+ InitializeAccessMethods();
+
/* initialize client encoding */
InitializeClientEncoding();
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 00018abb7d..df2a985900 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -30,6 +30,7 @@
#include <unistd.h>
#include "access/commit_ts.h"
+#include "access/compressamapi.h"
#include "access/gin.h"
#include "access/rmgr.h"
#include "access/tableam.h"
@@ -3915,6 +3916,17 @@ static struct config_string ConfigureNamesString[] =
check_default_table_access_method, NULL, NULL
},
+ {
+ {"default_toast_compression", PGC_USERSET, CLIENT_CONN_STATEMENT,
+ gettext_noop("Sets the default compression for new columns."),
+ NULL,
+ GUC_IS_NAME
+ },
+ &default_toast_compression,
+ DEFAULT_TOAST_COMPRESSION,
+ check_default_toast_compression, assign_default_toast_compression, NULL, NULL
+ },
+
{
{"default_tablespace", PGC_USERSET, CLIENT_CONN_STATEMENT,
gettext_noop("Sets the default tablespace to create tables and indexes in."),
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index d357ebb559..1513cafcf4 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -287,4 +287,6 @@ typedef struct IndexAmRoutine
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
+void InitializeAccessMethods(void);
+
#endif /* AMAPI_H */
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index 5a8e23d926..d75a8e9df2 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -17,6 +17,7 @@
#include "catalog/pg_am_d.h"
#include "nodes/nodes.h"
+#include "utils/guc.h"
/*
* Built-in compression method-id. The toast compression header will store
@@ -29,8 +30,17 @@ typedef enum CompressionId
LZ4_COMPRESSION_ID = 1
} CompressionId;
-/* Use default compression method if it is not specified. */
+/* Default compression method if not specified. */
+#define DEFAULT_TOAST_COMPRESSION "pglz"
#define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID
+
+/* GUC */
+extern char *default_toast_compression;
+extern void assign_default_toast_compression(const char *newval, void *extra);
+extern bool check_default_toast_compression(char **newval, void **extra, GucSource source);
+
+extern Oid GetDefaultToastCompression(void);
+
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
--
2.17.0
[text/x-diff] 0005-default-to-with-lz4.patch (1.7K, ../20210218211314.GC14772@telsasoft.com/6-0005-default-to-with-lz4.patch)
download | inline diff:
From 3c48e28acf33864852ba6f9c2e276af2c62e474b Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sat, 13 Feb 2021 22:11:24 -0600
Subject: [PATCH 05/12] default to --with-lz4
this is meant to excercize the new feature in the CIs, and not meant to be merged
---
configure | 6 ++++--
configure.ac | 4 ++--
2 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/configure b/configure
index 0895b2f326..79e9d226ff 100755
--- a/configure
+++ b/configure
@@ -1571,7 +1571,7 @@ Optional Packages:
--with-system-tzdata=DIR
use system time zone data in DIR
--without-zlib do not use Zlib
- --with-lz4 build with LZ4 support
+ --without-lz4 build without LZ4 support
--with-gnu-ld assume the C compiler uses GNU ld [default=no]
--with-ssl=LIB use LIB for SSL/TLS support (openssl)
--with-openssl obsolete spelling of --with-ssl=openssl
@@ -8592,7 +8592,9 @@ $as_echo "#define USE_LZ4 1" >>confdefs.h
esac
else
- with_lz4=no
+ with_lz4=yes
+
+$as_echo "#define USE_LZ4 1" >>confdefs.h
fi
diff --git a/configure.ac b/configure.ac
index 63940b78a0..62180c5424 100644
--- a/configure.ac
+++ b/configure.ac
@@ -990,8 +990,8 @@ AC_SUBST(with_zlib)
# LZ4
#
AC_MSG_CHECKING([whether to build with LZ4 support])
-PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support],
- [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])])
+PGAC_ARG_BOOL(with, lz4, yes, [build without LZ4 support],
+ [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build without LZ4 support. (--without-lz4)])])
AC_SUBST(with_lz4)
#
--
2.17.0
[text/x-diff] 0006-fixups.patch.patch (6.4K, ../20210218211314.GC14772@telsasoft.com/7-0006-fixups.patch.patch)
download | inline diff:
From 75814aed18a79c6774bf5e522cedc2a0d4f0d263 Mon Sep 17 00:00:00 2001
From: Robert Haas <robertmhaas@gmail.com>
Date: Wed, 10 Feb 2021 16:56:17 -0500
Subject: [PATCH 06/12] fixups.patch
---
doc/src/sgml/ddl.sgml | 3 ---
doc/src/sgml/ref/create_table.sgml | 15 +++++++++++----
src/backend/access/common/detoast.c | 5 ++---
src/backend/access/compression/compress_lz4.c | 10 +++++-----
src/backend/executor/nodeModifyTable.c | 2 +-
src/include/nodes/execnodes.h | 4 ++--
6 files changed, 21 insertions(+), 18 deletions(-)
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 4d7ed698b9..1e9a4625cc 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -3762,9 +3762,6 @@ CREATE TABLE measurement (
<productname>PostgreSQL</productname>
tables (or, possibly, foreign tables). It is possible to specify a
tablespace and storage parameters for each partition separately.
- By default, each column in a partition inherits the compression method
- from parent table's column, however a different compression method can be
- set for each partition.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index 3374012940..807597e648 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -997,10 +997,17 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term>
<listitem>
<para>
- This sets the compression method for a column. The supported compression
- methods are <literal>pglz</literal> and <literal>lz4</literal>.
- <literal>lz4</literal> is available only if <literal>--with-lz4</literal>
- was used when building <productname>PostgreSQL</productname>. The default
+ The <literal>COMPRESSION</literal> clause sets the compression method
+ for a column. Compression is supported only for variable-width data
+ types, and is used only for columns whose storage type is main or
+ extended. (See <xref linkend="sql-altertable"/> for information on
+ column storage types.) Setting this property for a partitioned table
+ has no direct effect, because such tables have no storage of their own,
+ but the configured value is inherited by newly-created partitions.
+ The supported compression methods are <literal>pglz</literal> and
+ <literal>lz4</literal>. <literal>lz4</literal> is available only if
+ <literal>--with-lz4</literal> was used when building
+ <productname>PostgreSQL</productname>. The default
is <literal>pglz</literal>.
</para>
</listitem>
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index b78d49167b..95d5b1c12a 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -477,8 +477,8 @@ toast_get_compression_oid(struct varlena *attr)
return InvalidOid;
/*
- * Just fetch the toast compress header to know the compression method
- * in the compressed data.
+ * Fetch just enough of the value to examine the compression header,
+ * so that we can find out the compression method.
*/
attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ_COMPRESS);
}
@@ -503,7 +503,6 @@ toast_get_compression_handler(struct varlena *attr)
cmid = TOAST_COMPRESS_METHOD(attr);
- /* Get the handler routines for the compression method */
switch (cmid)
{
case PGLZ_COMPRESSION_ID:
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 1856cf7df7..3079eff7eb 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -23,7 +23,7 @@
/*
* lz4_cmcompress - compression routine for lz4 compression method
*
- * Compresses source into dest using the default strategy. Returns the
+ * Compresses source into dest using the LZ4 defaults. Returns the
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
@@ -42,8 +42,8 @@ lz4_cmcompress(const struct varlena *value)
valsize = VARSIZE_ANY_EXHDR(value);
/*
- * Get maximum size of the compressed data that lz4 compression may output
- * and allocate the memory for holding the compressed data and the header.
+ * Figure out the maximum possible size of the LZ4 output, add the bytes
+ * that will be needed for varlena overhead, and allocate that amount.
*/
max_size = LZ4_compressBound(valsize);
tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
@@ -83,10 +83,10 @@ lz4_cmdecompress(const struct varlena *value)
int32 rawsize;
struct varlena *result;
- /* allocate memory for holding the uncompressed data */
+ /* allocate memory for the uncompressed data */
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
- /* decompress data using lz4 routine */
+ /* decompress the data */
rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
VARDATA(result),
VARSIZE(value) - VARHDRSZ_COMPRESS,
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index f77deee399..920d9dd0d5 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -2117,7 +2117,7 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
TupleTableSlot *newslot = *outslot;
/*
- * If the called has passed an invalid slot then create a new slot.
+ * If the caller has passed an invalid slot then create a new slot.
* Otherwise, just clear the existing tuple from the slot. This slot
* should be stored by the caller so that it can be reused for
* decompressing the subsequent tuples.
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index ac3884b3a8..182b77a0ff 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1186,8 +1186,8 @@ typedef struct ModifyTableState
TupleTableSlot *mt_root_tuple_slot;
/*
- * Slot for storing the modified tuple, incase the target attribute's
- * compression method doesn't match with the source table.
+ * Slot for storing the modified tuple, in case the target attribute's
+ * compression method doesn't match that of the source table.
*/
TupleTableSlot *mt_decompress_tuple_slot;
--
2.17.0
[text/x-diff] 0007-alter-table-set-compression.patch (29.3K, ../20210218211314.GC14772@telsasoft.com/8-0007-alter-table-set-compression.patch)
download | inline diff:
From 3a19bc428db37772f867de791ae0d92dd59a4866 Mon Sep 17 00:00:00 2001
From: dilipkumar <dilipbalaut@gmail.com>
Date: Fri, 18 Dec 2020 11:31:22 +0530
Subject: [PATCH 07/12] alter table set compression
Add support for changing the compression method associated
with a column, forcing a table rewrite.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Robert Haas and Tomas Vondra.
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov
Discussions:
https://www.postgresql.org/message-id/20171213151818.75a20259@postgrespro.ru
https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40mail.gmail.com
https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail.gmail.com
https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development
https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail.gmail.com
---
doc/src/sgml/ref/alter_table.sgml | 16 ++
src/backend/commands/createas.c | 3 +-
src/backend/commands/matview.c | 3 +-
src/backend/commands/tablecmds.c | 226 +++++++++++++++-----
src/backend/executor/nodeModifyTable.c | 9 +-
src/backend/parser/gram.y | 9 +
src/bin/psql/tab-complete.c | 2 +-
src/include/commands/event_trigger.h | 1 +
src/include/executor/executor.h | 3 +-
src/include/nodes/parsenodes.h | 3 +-
src/test/regress/expected/compression.out | 93 ++++++--
src/test/regress/expected/compression_1.out | 56 ++++-
src/test/regress/sql/compression.sql | 21 ++
13 files changed, 370 insertions(+), 75 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c25ef5abd6..0bd0c1a503 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,6 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -103,6 +104,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] |
UNIQUE <replaceable class="parameter">index_parameters</replaceable> |
PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> |
+ COMPRESSION <replaceable class="parameter">compression_method</replaceable> |
REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ]
[ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] }
[ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -383,6 +385,20 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ </term>
+ <listitem>
+ <para>
+ This sets the compression method for a column. The supported compression
+ methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ <literal>lz4</literal> is available only if <literal>--with-lz4</literal>
+ was used when building <productname>PostgreSQL</productname>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]</literal></term>
<listitem>
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index 1d17dc0d6b..748ec29570 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -589,7 +589,8 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self)
*/
slot = CompareCompressionMethodAndDecompress(slot,
&myState->decompress_tuple_slot,
- myState->rel->rd_att);
+ myState->rel->rd_att,
+ NULL);
/*
* Note that the input slot might not be of the type of the target
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index 713fc3fceb..779b4e51cf 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -494,7 +494,8 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self)
*/
slot = CompareCompressionMethodAndDecompress(slot,
&myState->decompress_tuple_slot,
- myState->transientrel->rd_att);
+ myState->transientrel->rd_att,
+ NULL);
/*
* Note that the input slot might not be of the type of the target
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 7adeaedd0e..5e24af9cb3 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -529,6 +529,8 @@ static void ATExecGenericOptions(Relation rel, List *options);
static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
+static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
+ const char *column, Node *newValue, LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -3968,6 +3970,7 @@ AlterTableGetLockLevel(List *cmds)
*/
case AT_GenericOptions:
case AT_AlterColumnGenericOptions:
+ case AT_SetCompression:
cmd_lockmode = AccessExclusiveLock;
break;
@@ -4495,7 +4498,8 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,
case AT_DisableRowSecurity:
case AT_ForceRowSecurity:
case AT_NoForceRowSecurity:
- ATSimplePermissions(rel, ATT_TABLE);
+ case AT_SetCompression:
+ ATSimplePermissions(rel, ATT_TABLE | ATT_MATVIEW);
/* These commands never recurse */
/* No command-specific prep needed */
pass = AT_PASS_MISC;
@@ -4903,6 +4907,10 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
Assert(rel->rd_rel->relkind == RELKIND_INDEX);
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
+ case AT_SetCompression:
+ address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ lockmode);
+ break;
default: /* oops */
elog(ERROR, "unrecognized alter table type: %d",
(int) cmd->subtype);
@@ -5519,6 +5527,7 @@ ATRewriteTable(AlteredTableInfo *tab, Oid OIDNewHeap, LOCKMODE lockmode)
while (table_scan_getnextslot(scan, ForwardScanDirection, oldslot))
{
+ bool decompressed = false;
TupleTableSlot *insertslot;
if (tab->rewrite > 0)
@@ -5527,11 +5536,25 @@ ATRewriteTable(AlteredTableInfo *tab, Oid OIDNewHeap, LOCKMODE lockmode)
slot_getallattrs(oldslot);
ExecClearTuple(newslot);
- /* copy attributes */
- memcpy(newslot->tts_values, oldslot->tts_values,
- sizeof(Datum) * oldslot->tts_nvalid);
- memcpy(newslot->tts_isnull, oldslot->tts_isnull,
- sizeof(bool) * oldslot->tts_nvalid);
+ if (tab->rewrite & AT_REWRITE_ALTER_COMPRESSION)
+ (void) CompareCompressionMethodAndDecompress(oldslot,
+ &newslot,
+ newTupDesc,
+ &decompressed);
+
+ /*
+ * copy attributes, if we have decompressed some attribute then
+ * the values and nulls array is already copied
+ */
+ if (!decompressed)
+ {
+ memcpy(newslot->tts_values, oldslot->tts_values,
+ sizeof(Datum) * oldslot->tts_nvalid);
+ memcpy(newslot->tts_isnull, oldslot->tts_isnull,
+ sizeof(bool) * oldslot->tts_nvalid);
+ }
+ else
+ ExecClearTuple(newslot);
/* Set dropped attributes to null in new tuple */
foreach(lc, dropped_attrs)
@@ -7767,6 +7790,67 @@ ATExecSetOptions(Relation rel, const char *colName, Node *options,
return address;
}
+/*
+ * Helper function for ATExecSetStorage and ATExecSetCompression
+ *
+ * Set the attcompression and attstorage for the respective index attribute if
+ * the respective input values are valid.
+ */
+static void
+ApplyChangesToIndexes(Relation rel, Relation attrelation, AttrNumber attnum,
+ Oid newcompression, char newstorage, LOCKMODE lockmode)
+{
+ HeapTuple tuple;
+ ListCell *lc;
+ Form_pg_attribute attrtuple;
+
+ foreach(lc, RelationGetIndexList(rel))
+ {
+ Oid indexoid = lfirst_oid(lc);
+ Relation indrel;
+ AttrNumber indattnum = 0;
+
+ indrel = index_open(indexoid, lockmode);
+
+ for (int i = 0; i < indrel->rd_index->indnatts; i++)
+ {
+ if (indrel->rd_index->indkey.values[i] == attnum)
+ {
+ indattnum = i + 1;
+ break;
+ }
+ }
+
+ if (indattnum == 0)
+ {
+ index_close(indrel, lockmode);
+ continue;
+ }
+
+ tuple = SearchSysCacheCopyAttNum(RelationGetRelid(indrel), indattnum);
+
+ if (HeapTupleIsValid(tuple))
+ {
+ attrtuple = (Form_pg_attribute) GETSTRUCT(tuple);
+
+ if (OidIsValid(newcompression))
+ attrtuple->attcompression = newcompression;
+
+ if (newstorage != '\0')
+ attrtuple->attstorage = newstorage;
+
+ CatalogTupleUpdate(attrelation, &tuple->t_self, tuple);
+
+ InvokeObjectPostAlterHook(RelationRelationId,
+ RelationGetRelid(rel),
+ attrtuple->attnum);
+
+ heap_freetuple(tuple);
+ }
+
+ index_close(indrel, lockmode);
+ }
+}
/*
* ALTER TABLE ALTER COLUMN SET STORAGE
*
@@ -7782,7 +7866,6 @@ ATExecSetStorage(Relation rel, const char *colName, Node *newValue, LOCKMODE loc
Form_pg_attribute attrtuple;
AttrNumber attnum;
ObjectAddress address;
- ListCell *lc;
Assert(IsA(newValue, String));
storagemode = strVal(newValue);
@@ -7846,47 +7929,8 @@ ATExecSetStorage(Relation rel, const char *colName, Node *newValue, LOCKMODE loc
* Apply the change to indexes as well (only for simple index columns,
* matching behavior of index.c ConstructTupleDescriptor()).
*/
- foreach(lc, RelationGetIndexList(rel))
- {
- Oid indexoid = lfirst_oid(lc);
- Relation indrel;
- AttrNumber indattnum = 0;
-
- indrel = index_open(indexoid, lockmode);
-
- for (int i = 0; i < indrel->rd_index->indnatts; i++)
- {
- if (indrel->rd_index->indkey.values[i] == attnum)
- {
- indattnum = i + 1;
- break;
- }
- }
-
- if (indattnum == 0)
- {
- index_close(indrel, lockmode);
- continue;
- }
-
- tuple = SearchSysCacheCopyAttNum(RelationGetRelid(indrel), indattnum);
-
- if (HeapTupleIsValid(tuple))
- {
- attrtuple = (Form_pg_attribute) GETSTRUCT(tuple);
- attrtuple->attstorage = newstorage;
-
- CatalogTupleUpdate(attrelation, &tuple->t_self, tuple);
-
- InvokeObjectPostAlterHook(RelationRelationId,
- RelationGetRelid(rel),
- attrtuple->attnum);
-
- heap_freetuple(tuple);
- }
-
- index_close(indrel, lockmode);
- }
+ ApplyChangesToIndexes(rel, attrelation, attnum, InvalidOid, newstorage,
+ lockmode);
table_close(attrelation, RowExclusiveLock);
@@ -15033,6 +15077,92 @@ ATExecGenericOptions(Relation rel, List *options)
heap_freetuple(tuple);
}
+/*
+ * ALTER TABLE ALTER COLUMN SET COMPRESSION
+ *
+ * Return value is the address of the modified column
+ */
+static ObjectAddress
+ATExecSetCompression(AlteredTableInfo *tab,
+ Relation rel,
+ const char *column,
+ Node *newValue,
+ LOCKMODE lockmode)
+{
+ Relation attrel;
+ HeapTuple tuple;
+ Form_pg_attribute atttableform;
+ AttrNumber attnum;
+ char *compression;
+ char typstorage;
+ Oid cmoid;
+ Datum values[Natts_pg_attribute];
+ bool nulls[Natts_pg_attribute];
+ bool replace[Natts_pg_attribute];
+ ObjectAddress address;
+
+ Assert(IsA(newValue, String));
+ compression = strVal(newValue);
+
+ attrel = table_open(AttributeRelationId, RowExclusiveLock);
+
+ tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
+ if (!HeapTupleIsValid(tuple))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column \"%s\" of relation \"%s\" does not exist",
+ column, RelationGetRelationName(rel))));
+
+ /* Prevent them from altering a system attribute */
+ atttableform = (Form_pg_attribute) GETSTRUCT(tuple);
+ attnum = atttableform->attnum;
+ if (attnum <= 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot alter system column \"%s\"", column)));
+
+ typstorage = get_typstorage(atttableform->atttypid);
+
+ /* Prevent from setting compression methods for uncompressible type */
+ if (!IsStorageCompressible(typstorage))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(atttableform->atttypid))));
+
+ /* Initialize buffers for new tuple values */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replace, false, sizeof(replace));
+
+ /* Get the attribute compression method. */
+ cmoid = GetAttributeCompression(atttableform, compression);
+
+ if (atttableform->attcompression != cmoid)
+ tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+
+ atttableform->attcompression = cmoid;
+ CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
+
+ InvokeObjectPostAlterHook(RelationRelationId,
+ RelationGetRelid(rel),
+ atttableform->attnum);
+
+ ReleaseSysCache(tuple);
+
+ /* apply changes to the index column as well */
+ ApplyChangesToIndexes(rel, attrel, attnum, cmoid, '\0', lockmode);
+ table_close(attrel, RowExclusiveLock);
+
+ /* make changes visible */
+ CommandCounterIncrement();
+
+ ObjectAddressSubSet(address, RelationRelationId,
+ RelationGetRelid(rel), atttableform->attnum);
+ return address;
+}
+
+
/*
* Preparation phase for SET LOGGED/UNLOGGED
*
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index 920d9dd0d5..ea82a05591 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -2052,7 +2052,8 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate,
TupleTableSlot *
CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
TupleTableSlot **outslot,
- TupleDesc targetTupDesc)
+ TupleDesc targetTupDesc,
+ bool *decompressed)
{
int i;
int attnum;
@@ -2139,6 +2140,9 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
*outslot = newslot;
+ if (decompressed != NULL)
+ *decompressed = true;
+
return newslot;
}
@@ -2360,7 +2364,8 @@ ExecModifyTable(PlanState *pstate)
*/
slot = CompareCompressionMethodAndDecompress(slot,
&node->mt_decompress_tuple_slot,
- resultRelInfo->ri_RelationDesc->rd_att);
+ resultRelInfo->ri_RelationDesc->rd_att,
+ NULL);
switch (operation)
{
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 52d92df25d..30acfe615d 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -2308,6 +2308,15 @@ alter_table_cmd:
n->missing_ok = true;
$$ = (Node *)n;
}
+ /* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
+ | ALTER opt_column ColId SET optColumnCompression
+ {
+ AlterTableCmd *n = makeNode(AlterTableCmd);
+ n->subtype = AT_SetCompression;
+ n->name = $3;
+ n->def = (Node *) makeString($5);
+ $$ = (Node *)n;
+ }
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
| DROP opt_column IF_P EXISTS ColId opt_drop_behavior
{
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index b64db82f02..2aedddc1d8 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2107,7 +2107,7 @@ psql_completion(const char *text, int start, int end)
/* ALTER TABLE ALTER [COLUMN] <foo> SET */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET"))
- COMPLETE_WITH("(", "DEFAULT", "NOT NULL", "STATISTICS", "STORAGE");
+ COMPLETE_WITH("(", "COMPRESSION", "DEFAULT", "NOT NULL", "STATISTICS", "STORAGE");
/* ALTER TABLE ALTER [COLUMN] <foo> SET ( */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
diff --git a/src/include/commands/event_trigger.h b/src/include/commands/event_trigger.h
index c11bf2d781..5a314f4c1d 100644
--- a/src/include/commands/event_trigger.h
+++ b/src/include/commands/event_trigger.h
@@ -32,6 +32,7 @@ typedef struct EventTriggerData
#define AT_REWRITE_ALTER_PERSISTENCE 0x01
#define AT_REWRITE_DEFAULT_VAL 0x02
#define AT_REWRITE_COLUMN_REWRITE 0x04
+#define AT_REWRITE_ALTER_COMPRESSION 0x08
/*
* EventTriggerData is the node type that is passed as fmgr "context" info
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index 6495162a33..050ef2dcd0 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -623,5 +623,6 @@ extern void CheckSubscriptionRelkind(char relkind, const char *nspname,
const char *relname);
extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
TupleTableSlot **outslot,
- TupleDesc targetTupDesc);
+ TupleDesc targetTupDesc,
+ bool *decompressed);
#endif /* EXECUTOR_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 19d2ba26bf..f9a87dee02 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1901,7 +1901,8 @@ typedef enum AlterTableType
AT_AddIdentity, /* ADD IDENTITY */
AT_SetIdentity, /* SET identity column options */
AT_DropIdentity, /* DROP IDENTITY */
- AT_AlterCollationRefreshVersion /* ALTER COLLATION ... REFRESH VERSION */
+ AT_AlterCollationRefreshVersion, /* ALTER COLLATION ... REFRESH VERSION */
+ AT_SetCompression /* SET COMPRESSION */
} AlterTableType;
typedef struct ReplicaIdentityStmt
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 167878e78b..21c1b451d2 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -4,20 +4,20 @@ CREATE TABLE cmdata(f1 text COMPRESSION pglz);
CREATE INDEX idx ON cmdata(f1);
INSERT INTO cmdata VALUES(repeat('1234567890',1000));
\d+ cmdata
- Table "public.cmdata"
- Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
---------+------+-----------+----------+---------+----------+--------------+-------------
- f1 | text | | | | extended | |
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
Indexes:
"idx" btree (f1)
CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4);
INSERT INTO cmdata1 VALUES(repeat('1234567890',1004));
\d+ cmdata1
- Table "public.cmdata1"
- Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
---------+------+-----------+----------+---------+----------+--------------+-------------
- f1 | text | | | | extended | |
+ Table "public.cmdata1"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
-- try setting compression for incompressible data type
CREATE TABLE cmdata2 (f1 int COMPRESSION pglz);
@@ -112,18 +112,18 @@ DROP TABLE cmdata2;
-- test LIKE INCLUDING COMPRESSION
CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION);
\d+ cmdata2
- Table "public.cmdata2"
- Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
---------+------+-----------+----------+---------+----------+--------------+-------------
- f1 | text | | | | extended | |
+ Table "public.cmdata2"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
-- test compression with materialized view
CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1;
\d+ mv
- Materialized view "public.mv"
- Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
---------+------+-----------+----------+---------+----------+--------------+-------------
- x | text | | | | extended | |
+ Materialized view "public.mv"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ x | text | | | | extended | pglz | |
View definition:
SELECT cmdata1.f1 AS x
FROM cmdata1;
@@ -165,6 +165,67 @@ CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata);
NOTICE: merging column "f1" with inherited definition
ERROR: column "f1" has a compression method conflict
DETAIL: pglz versus lz4
+-- test alter compression method with rewrite
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4;
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+(1 row)
+
+-- test alter compression method for the materialized view
+ALTER TABLE cmdata1 ALTER COLUMN f1 SET COMPRESSION pglz;
+ALTER MATERIALIZED VIEW mv ALTER COLUMN x SET COMPRESSION lz4;
+REFRESH MATERIALIZED VIEW mv;
+\d+ mv
+ Materialized view "public.mv"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ x | text | | | | extended | lz4 | |
+View definition:
+ SELECT cmdata1.f1 AS x
+ FROM cmdata1;
+
+SELECT pg_column_compression(f1) FROM cmdata1;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+SELECT pg_column_compression(x) FROM mv;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+(2 rows)
+
+-- test alter compression method for the partitioned table
+ALTER TABLE cmpart ALTER COLUMN f1 SET COMPRESSION pglz;
+SELECT pg_column_compression(f1) FROM cmpart;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
+ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
+SELECT pg_column_compression(f1) FROM cmpart;
+ pg_column_compression
+-----------------------
+ pglz
+ lz4
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 329e7881b0..64c5855bf7 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -4,10 +4,10 @@ CREATE TABLE cmdata(f1 text COMPRESSION pglz);
CREATE INDEX idx ON cmdata(f1);
INSERT INTO cmdata VALUES(repeat('1234567890',1000));
\d+ cmdata
- Table "public.cmdata"
- Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
---------+------+-----------+----------+---------+----------+--------------+-------------
- f1 | text | | | | extended | |
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
Indexes:
"idx" btree (f1)
@@ -157,6 +157,54 @@ CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata);
NOTICE: merging column "f1" with inherited definition
ERROR: column "f1" has a compression method conflict
DETAIL: pglz versus lz4
+-- test alter compression method with rewrite
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4;
+ERROR: not built with lz4 support
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+(1 row)
+
+-- test alter compression method for the materialized view
+ALTER TABLE cmdata1 ALTER COLUMN f1 SET COMPRESSION pglz;
+ERROR: relation "cmdata1" does not exist
+ALTER MATERIALIZED VIEW mv ALTER COLUMN x SET COMPRESSION lz4;
+ERROR: relation "mv" does not exist
+REFRESH MATERIALIZED VIEW mv;
+ERROR: relation "mv" does not exist
+\d+ mv
+SELECT pg_column_compression(f1) FROM cmdata1;
+ERROR: relation "cmdata1" does not exist
+LINE 1: SELECT pg_column_compression(f1) FROM cmdata1;
+ ^
+SELECT pg_column_compression(x) FROM mv;
+ERROR: relation "mv" does not exist
+LINE 1: SELECT pg_column_compression(x) FROM mv;
+ ^
+-- test alter compression method for the partitioned table
+ALTER TABLE cmpart ALTER COLUMN f1 SET COMPRESSION pglz;
+ERROR: relation "cmpart" does not exist
+SELECT pg_column_compression(f1) FROM cmpart;
+ERROR: relation "cmpart" does not exist
+LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
+ ^
+ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
+ERROR: relation "cmpart1" does not exist
+ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmpart;
+ERROR: relation "cmpart" does not exist
+LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
+ ^
-- check data is ok
SELECT length(f1) FROM cmdata;
length
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 450416ecb4..b9daa33b74 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -75,6 +75,27 @@ SELECT pg_column_compression(f1) FROM cmpart;
CREATE TABLE cminh() INHERITS(cmdata, cmdata1);
CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata);
+-- test alter compression method with rewrite
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4;
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+
+-- test alter compression method for the materialized view
+ALTER TABLE cmdata1 ALTER COLUMN f1 SET COMPRESSION pglz;
+ALTER MATERIALIZED VIEW mv ALTER COLUMN x SET COMPRESSION lz4;
+REFRESH MATERIALIZED VIEW mv;
+\d+ mv
+SELECT pg_column_compression(f1) FROM cmdata1;
+SELECT pg_column_compression(x) FROM mv;
+
+-- test alter compression method for the partitioned table
+ALTER TABLE cmpart ALTER COLUMN f1 SET COMPRESSION pglz;
+SELECT pg_column_compression(f1) FROM cmpart;
+
+ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
+ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
+SELECT pg_column_compression(f1) FROM cmpart;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
[text/x-diff] 0008-Add-support-for-PRESERVE.patch (52.4K, ../20210218211314.GC14772@telsasoft.com/9-0008-Add-support-for-PRESERVE.patch)
download | inline diff:
From f6cde260dc4350bb54cb48f078cc1d07eac0bb63 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilipkumar@localhost.localdomain>
Date: Wed, 10 Feb 2021 11:49:23 +0530
Subject: [PATCH 08/12] Add support for PRESERVE
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Now the compression method can be changed without forcing a table
rewrite, by including the old method in the PRESERVE list. Â For
supporting this the column will maintain the dependency with all
the supported compression methods. Â So whenever the compression
method is altered the dependency is added with the new compression
method and the dependency is removed for all the old compression
methods which are not given in the preserve list. Â If PRESERVE ALL
is given then all the dependency is maintained.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 10 +-
src/backend/catalog/pg_depend.c | 7 +
src/backend/commands/Makefile | 1 +
src/backend/commands/compressioncmds.c | 300 ++++++++++++++++++++
src/backend/commands/tablecmds.c | 126 ++++----
src/backend/executor/nodeModifyTable.c | 12 +-
src/backend/nodes/copyfuncs.c | 17 +-
src/backend/nodes/equalfuncs.c | 15 +-
src/backend/nodes/outfuncs.c | 15 +-
src/backend/parser/gram.y | 52 +++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 101 +++++++
src/bin/pg_dump/pg_dump.h | 15 +-
src/bin/psql/tab-complete.c | 7 +
src/include/commands/defrem.h | 7 +
src/include/commands/tablecmds.h | 2 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 16 +-
src/test/regress/expected/compression.out | 37 ++-
src/test/regress/expected/compression_1.out | 39 ++-
src/test/regress/expected/create_index.out | 56 ++--
src/test/regress/sql/compression.sql | 9 +
22 files changed, 739 insertions(+), 108 deletions(-)
create mode 100644 src/backend/commands/compressioncmds.c
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 0bd0c1a503..c9f443a59c 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable>
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -387,7 +387,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
<varlistentry>
<term>
- <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal>
+ <literal>SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]</literal>
</term>
<listitem>
<para>
@@ -395,6 +395,12 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
+ The <literal>PRESERVE</literal> list contains a list of compression
+ methods used in the column and determines which of them may be kept.
+ Without <literal>PRESERVE</literal> or if any of the pre-existing
+ compression methods are not preserved, the table will be rewritten. If
+ <literal>PRESERVE ALL</literal> is specified, then all of the existing
+ methods will be preserved and the table will not be rewritten.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c
index 63da24322d..dd376484b7 100644
--- a/src/backend/catalog/pg_depend.c
+++ b/src/backend/catalog/pg_depend.c
@@ -17,6 +17,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
#include "access/table.h"
+#include "catalog/pg_am.h"
#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_collation.h"
@@ -125,6 +126,12 @@ recordMultipleDependencies(const ObjectAddress *depender,
if (referenced->objectId == DEFAULT_COLLATION_OID)
ignore_systempin = true;
}
+ /*
+ * Record the dependency on compression access method for handling
+ * preserve.
+ */
+ if (referenced->classId == AccessMethodRelationId)
+ ignore_systempin = true;
}
else
Assert(!version);
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index e8504f0ae4..a7395ad77d 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -21,6 +21,7 @@ OBJS = \
cluster.o \
collationcmds.o \
comment.o \
+ compressioncmds.o \
constraint.o \
conversioncmds.o \
copy.o \
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
new file mode 100644
index 0000000000..fd6db24e7f
--- /dev/null
+++ b/src/backend/commands/compressioncmds.c
@@ -0,0 +1,300 @@
+/*-------------------------------------------------------------------------
+ *
+ * compressioncmds.c
+ * Routines for SQL commands for attribute compression methods
+ *
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/compressioncmds.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/compressamapi.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "catalog/catalog.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_attribute.h"
+#include "catalog/pg_depend.h"
+#include "commands/defrem.h"
+#include "commands/tablecmds.h"
+#include "miscadmin.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+
+/*
+ * Get list of all supported compression methods for the given attribute.
+ *
+ * We maintain dependency of the attribute on the pg_am row for the current
+ * compression AM and all the preserved compression AM. So scan pg_depend and
+ * find the column dependency on the pg_am. Collect the list of access method
+ * oids on which this attribute has a dependency.
+ */
+static List *
+lookup_attribute_compression(Oid attrelid, AttrNumber attnum, List *oldcmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+ List *cmoids = NIL;
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (depform->refclassid == AccessMethodRelationId)
+ cmoids = list_append_unique_oid(cmoids, depform->refobjid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+
+ return cmoids;
+}
+
+/*
+ * Remove the attribute dependency on the old compression methods
+ *
+ * Scan the pg_depend and search the attribute dependency on the pg_am. Remove
+ * dependency on previous am which is not preserved. The list of non-preserved
+ * AMs is given in cmoids.
+ */
+static void
+remove_old_dependencies(Oid attrelid, AttrNumber attnum, List *cmoids)
+{
+ LOCKMODE lock = AccessShareLock;
+ HeapTuple tup;
+ Relation rel;
+ SysScanDesc scan;
+ ScanKeyData key[3];
+
+ rel = table_open(DependRelationId, lock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_depend_classid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationRelationId));
+ ScanKeyInit(&key[1],
+ Anum_pg_depend_objid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(attrelid));
+ ScanKeyInit(&key[2],
+ Anum_pg_depend_objsubid,
+ BTEqualStrategyNumber, F_INT4EQ,
+ Int32GetDatum((int32) attnum));
+
+ scan = systable_beginscan(rel, DependDependerIndexId, true,
+ NULL, 3, key);
+
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup);
+
+ if (list_member_oid(cmoids, depform->refobjid))
+ {
+ Assert(depform->refclassid == AccessMethodRelationId);
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(rel, lock);
+}
+
+/*
+ * Check whether the given compression method oid is supported by
+ * the target attribute.
+ */
+bool
+IsCompressionSupported(Form_pg_attribute att, Oid cmoid)
+{
+ List *cmoids = NIL;
+
+ /* Check whether it is same as the current compression oid */
+ if (cmoid == att->attcompression)
+ return true;
+
+ /* Check the oid in all preserved compresion methods */
+ cmoids = lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+ if (list_member_oid(cmoids, cmoid))
+ return true;
+ else
+ return false;
+}
+
+/*
+ * In binary upgrade mode add the dependencies for all the preserved compression
+ * method.
+ */
+static void
+BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
+{
+ ListCell *cell;
+
+ foreach(cell, preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ add_column_compression_dependency(att->attrelid, att->attnum, cmoid_p);
+ }
+}
+
+/*
+ * Get the compression method oid based on the compression method name. When
+ * compression is not specified returns default attribute compression. It is
+ * possible case for CREATE TABLE and ADD COLUMN commands where COMPRESSION
+ * syntax is optional.
+ *
+ * For ALTER command, check all the supported compression methods for the
+ * attribute and if the preserve list is not passed or some of the old
+ * compression methods are not given in the preserved list then delete
+ * dependency from the old compression methods and force the table rewrite.
+ */
+Oid
+GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
+ bool *need_rewrite)
+{
+ Oid cmoid;
+ char typstorage = get_typstorage(att->atttypid);
+ ListCell *cell;
+
+ /*
+ * No compression for the plain/external storage, refer comments atop
+ * attcompression parameter in pg_attribute.h
+ */
+ if (!IsStorageCompressible(typstorage))
+ {
+ if (compression == NULL)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("column data type %s does not support compression",
+ format_type_be(att->atttypid))));
+ }
+
+ /* fallback to default compression if it's not specified */
+ if (compression == NULL)
+ return GetDefaultToastCompression();
+
+ cmoid = get_compression_am_oid(compression->cmname, false);
+
+#ifndef HAVE_LIBLZ4
+ if (cmoid == LZ4_COMPRESSION_AM_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#endif
+
+ /*
+ * Determine if the column needs rewrite or not. Rewrite conditions: SET
+ * COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
+ * with full list of previous access methods.
+ */
+ if (need_rewrite != NULL)
+ {
+ List *previous_cmoids = NIL;
+
+ *need_rewrite = false;
+
+ /*
+ * In binary upgrade mode, just create a dependency on all preserved
+ * methods.
+ */
+ if (IsBinaryUpgrade)
+ {
+ BinaryUpgradeAddPreserve(att, compression->preserve);
+ return cmoid;
+ }
+
+ /* If we have preserved all then rewrite is not required */
+ if (compression->preserve_all)
+ return cmoid;
+
+ previous_cmoids =
+ lookup_attribute_compression(att->attrelid, att->attnum, NULL);
+
+ foreach(cell, compression->preserve)
+ {
+ char *cmname_p = strVal(lfirst(cell));
+ Oid cmoid_p = get_compression_am_oid(cmname_p, false);
+
+ if (!list_member_oid(previous_cmoids, cmoid_p))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("\"%s\" compression access method cannot be preserved", cmname_p)));
+
+ /*
+ * Remove from previous list, also protect from duplicate
+ * entries in the PRESERVE list
+ */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid_p);
+ }
+
+ /* delete the current cmoid from the list */
+ previous_cmoids = list_delete_oid(previous_cmoids, cmoid);
+
+ /*
+ * If the list of previous Oids is not empty after deletions then
+ * we need to rewrite tuples in the table. Also remove the dependency
+ * on the old compression methods which are no longer preserved.
+ */
+ if (list_length(previous_cmoids) != 0)
+ {
+ remove_old_dependencies(att->attrelid, att->attnum,
+ previous_cmoids);
+ *need_rewrite = true;
+ }
+
+ /* Cleanup */
+ list_free(previous_cmoids);
+ }
+
+ return cmoid;
+}
+
+/*
+ * Construct ColumnCompression node from the compression method oid.
+ */
+ColumnCompression *
+MakeColumnCompression(Oid attcompression)
+{
+ ColumnCompression *node;
+
+ if (!OidIsValid(attcompression))
+ return NULL;
+
+ node = makeNode(ColumnCompression);
+ node->cmname = get_am_name(attcompression);
+
+ return node;
+}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 5e24af9cb3..5fba7352a3 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -530,7 +530,9 @@ static void ATExecEnableRowSecurity(Relation rel);
static void ATExecDisableRowSecurity(Relation rel);
static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls);
static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel,
- const char *column, Node *newValue, LOCKMODE lockmode);
+ const char *column,
+ ColumnCompression *compression,
+ LOCKMODE lockmode);
static void index_copy_data(Relation rel, RelFileNode newrnode);
static const char *storage_name(char c);
@@ -562,7 +564,6 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx,
static List *GetParentedForeignKeyRefs(Relation partition);
static void ATDetachCheckNoForeignKeyRefs(Relation partition);
static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll);
-static Oid GetAttributeCompression(Form_pg_attribute att, char *compression);
/* ----------------------------------------------------------------
* DefineRelation
@@ -587,6 +588,7 @@ ObjectAddress
DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ObjectAddress *typaddress, const char *queryString)
{
+ int i;
char relname[NAMEDATALEN];
Oid namespaceId;
Oid relationId;
@@ -865,7 +867,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
attr->attcompression =
- GetAttributeCompression(attr, colDef->compression);
+ GetAttributeCompression(attr, colDef->compression, NULL);
else
attr->attcompression = InvalidOid;
}
@@ -935,6 +937,20 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
rel = relation_open(relationId, AccessExclusiveLock);
+ /*
+ * Add the dependency on the respective compression AM for the relation
+ * attributes.
+ */
+ for (i = 0; i < (RelationGetDescr(rel))->natts; i++)
+ {
+ Form_pg_attribute attr;
+
+ attr = TupleDescAttr(RelationGetDescr(rel), i);
+ if (OidIsValid(attr->attcompression))
+ add_column_compression_dependency(attr->attrelid, attr->attnum,
+ attr->attcompression);
+ }
+
/*
* Now add any newly specified column default and generation expressions
* to the new relation. These are passed to us in the form of raw
@@ -2415,16 +2431,17 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
/* Copy/check compression parameter */
if (OidIsValid(attribute->attcompression))
{
- char *compression = get_am_name(attribute->attcompression);
+ ColumnCompression *compression =
+ MakeColumnCompression(attribute->attcompression);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression, compression) != 0)
+ else if (strcmp(def->compression->cmname, compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, compression)));
+ errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
}
def->inhcount++;
@@ -2461,7 +2478,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(
+ attribute->attcompression);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2712,12 +2730,12 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->compression = newdef->compression;
else if (newdef->compression)
{
- if (strcmp(def->compression, newdef->compression))
+ if (strcmp(def->compression->cmname, newdef->compression->cmname))
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("column \"%s\" has a compression method conflict",
attributeName),
- errdetail("%s versus %s", def->compression, newdef->compression)));
+ errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
}
/* Mark the column as locally defined */
@@ -4908,7 +4926,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
ATExecAlterCollationRefreshVersion(rel, cmd->object);
break;
case AT_SetCompression:
- address = ATExecSetCompression(tab, rel, cmd->name, cmd->def,
+ address = ATExecSetCompression(tab, rel, cmd->name,
+ (ColumnCompression *) cmd->def,
lockmode);
break;
default: /* oops */
@@ -6414,7 +6433,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
if (rel->rd_rel->relkind == RELKIND_RELATION ||
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
- colDef->compression);
+ colDef->compression,
+ NULL);
else
attribute.attcompression = InvalidOid;
@@ -6589,6 +6609,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
*/
add_column_datatype_dependency(myrelid, newattnum, attribute.atttypid);
add_column_collation_dependency(myrelid, newattnum, attribute.attcollation);
+ add_column_compression_dependency(myrelid, newattnum, attribute.attcompression);
/*
* Propagate to children as appropriate. Unlike most other ALTER
@@ -6736,6 +6757,28 @@ add_column_collation_dependency(Oid relid, int32 attnum, Oid collid)
}
}
+/*
+ * Install a dependency for compression on its column.
+ *
+ * This is used for identifying all the supported compression methods
+ * (current and preserved) for a attribute.
+ *
+ * If dependency is already there the whole thing is skipped.
+ */
+void
+add_column_compression_dependency(Oid relid, int32 attnum, Oid cmoid)
+{
+ ObjectAddress acref,
+ attref;
+
+ Assert(relid > 0 && attnum > 0);
+
+ ObjectAddressSet(acref, AccessMethodRelationId, cmoid);
+ ObjectAddressSubSet(attref, RelationRelationId, relid, attnum);
+
+ recordMultipleDependencies(&attref, &acref, 1, DEPENDENCY_NORMAL, true);
+}
+
/*
* ALTER TABLE ALTER COLUMN DROP NOT NULL
*/
@@ -11867,7 +11910,8 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
foundDep->refobjid == attTup->attcollation) &&
!(foundDep->refclassid == RelationRelationId &&
foundDep->refobjid == RelationGetRelid(rel) &&
- foundDep->refobjsubid != 0)
+ foundDep->refobjsubid != 0) &&
+ foundDep->refclassid != AccessMethodRelationId
)
elog(ERROR, "found unexpected dependency for column: %s",
getObjectDescription(&foundObject, false));
@@ -11982,6 +12026,11 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel,
add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype);
add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid);
+ /* Create dependency for new attribute compression */
+ if (OidIsValid(attTup->attcompression))
+ add_column_compression_dependency(RelationGetRelid(rel), attnum,
+ attTup->attcompression);
+
/*
* Drop any pg_statistic entry for the column, since it's now wrong type
*/
@@ -15086,24 +15135,21 @@ static ObjectAddress
ATExecSetCompression(AlteredTableInfo *tab,
Relation rel,
const char *column,
- Node *newValue,
+ ColumnCompression *compression,
LOCKMODE lockmode)
{
Relation attrel;
HeapTuple tuple;
Form_pg_attribute atttableform;
AttrNumber attnum;
- char *compression;
char typstorage;
Oid cmoid;
+ bool need_rewrite;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
ObjectAddress address;
- Assert(IsA(newValue, String));
- compression = strVal(newValue);
-
attrel = table_open(AttributeRelationId, RowExclusiveLock);
tuple = SearchSysCacheAttName(RelationGetRelid(rel), column);
@@ -15136,11 +15182,16 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression);
+ cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
if (atttableform->attcompression != cmoid)
+ add_column_compression_dependency(atttableform->attrelid,
+ atttableform->attnum, cmoid);
+ if (need_rewrite)
tab->rewrite |= AT_REWRITE_ALTER_COMPRESSION;
+ atttableform->attcompression = cmoid;
+
atttableform->attcompression = cmoid;
CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
@@ -17865,42 +17916,3 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll)
index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false));
CacheInvalidateRelcache(rel);
}
-
-/*
- * resolve column compression specification to an OID.
- */
-static Oid
-GetAttributeCompression(Form_pg_attribute att, char *compression)
-{
- char typstorage = get_typstorage(att->atttypid);
- Oid amoid;
-
- /*
- * No compression for the plain/external storage, refer comments atop
- * attcompression parameter in pg_attribute.h
- */
- if (!IsStorageCompressible(typstorage))
- {
- if (compression == NULL)
- return InvalidOid;
-
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("column data type %s does not support compression",
- format_type_be(att->atttypid))));
- }
-
- /* fallback to default compression if it's not specified */
- if (compression == NULL)
- return GetDefaultToastCompression();
-
- amoid = get_compression_am_oid(compression, false);
-
-#ifndef HAVE_LIBLZ4
- if (amoid == LZ4_COMPRESSION_AM_OID)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("not built with lz4 support")));
-#endif
- return amoid;
-}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index ea82a05591..90d092671e 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -44,6 +44,7 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "catalog/catalog.h"
+#include "commands/defrem.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -2068,8 +2069,8 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
/*
* Loop over all the attributes in the tuple and check if any attribute is
- * compressed and its compression method is not same as the target
- * atrribute's compression method then decompress it.
+ * compressed and its compression method is not is not supported by the
+ * target attribute then we need to decompress
*/
for (i = 0; i < natts; i++)
{
@@ -2094,12 +2095,13 @@ CompareCompressionMethodAndDecompress(TupleTableSlot *slot,
DatumGetPointer(slot->tts_values[attnum - 1]);
/*
- * Get the compression method Oid stored in the toast header and
- * compare it with the compression method of the target.
+ * Get the compression method stored in the toast header and if the
+ * compression method is not supported by the target attribute then
+ * we need to decompress it.
*/
cmoid = toast_get_compression_oid(new_value);
if (OidIsValid(cmoid) &&
- targetTupDesc->attrs[i].attcompression != cmoid)
+ !IsCompressionSupported(&targetTupDesc->attrs[i], cmoid))
{
new_value = detoast_attr(new_value);
slot->tts_values[attnum - 1] = PointerGetDatum(new_value);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1338e04409..6a11f8eb60 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2966,7 +2966,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_STRING_FIELD(colname);
COPY_NODE_FIELD(typeName);
- COPY_STRING_FIELD(compression);
+ COPY_NODE_FIELD(compression);
COPY_SCALAR_FIELD(inhcount);
COPY_SCALAR_FIELD(is_local);
COPY_SCALAR_FIELD(is_not_null);
@@ -2986,6 +2986,18 @@ _copyColumnDef(const ColumnDef *from)
return newnode;
}
+static ColumnCompression *
+_copyColumnCompression(const ColumnCompression *from)
+{
+ ColumnCompression *newnode = makeNode(ColumnCompression);
+
+ COPY_STRING_FIELD(cmname);
+ COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(preserve);
+
+ return newnode;
+}
+
static Constraint *
_copyConstraint(const Constraint *from)
{
@@ -5675,6 +5687,9 @@ copyObjectImpl(const void *from)
case T_ColumnDef:
retval = _copyColumnDef(from);
break;
+ case T_ColumnCompression:
+ retval = _copyColumnCompression(from);
+ break;
case T_Constraint:
retval = _copyConstraint(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f3592003da..26a9b85974 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2599,7 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
{
COMPARE_STRING_FIELD(colname);
COMPARE_NODE_FIELD(typeName);
- COMPARE_STRING_FIELD(compression);
+ COMPARE_NODE_FIELD(compression);
COMPARE_SCALAR_FIELD(inhcount);
COMPARE_SCALAR_FIELD(is_local);
COMPARE_SCALAR_FIELD(is_not_null);
@@ -2619,6 +2619,16 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
return true;
}
+static bool
+_equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
+{
+ COMPARE_STRING_FIELD(cmname);
+ COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(preserve);
+
+ return true;
+}
+
static bool
_equalConstraint(const Constraint *a, const Constraint *b)
{
@@ -3724,6 +3734,9 @@ equal(const void *a, const void *b)
case T_ColumnDef:
retval = _equalColumnDef(a, b);
break;
+ case T_ColumnCompression:
+ retval = _equalColumnCompression(a, b);
+ break;
case T_Constraint:
retval = _equalConstraint(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 0605ef3f84..b584a58ba3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2863,7 +2863,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_STRING_FIELD(colname);
WRITE_NODE_FIELD(typeName);
- WRITE_STRING_FIELD(compression);
+ WRITE_NODE_FIELD(compression);
WRITE_INT_FIELD(inhcount);
WRITE_BOOL_FIELD(is_local);
WRITE_BOOL_FIELD(is_not_null);
@@ -2881,6 +2881,16 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_LOCATION_FIELD(location);
}
+static void
+_outColumnCompression(StringInfo str, const ColumnCompression *node)
+{
+ WRITE_NODE_TYPE("COLUMNCOMPRESSION");
+
+ WRITE_STRING_FIELD(cmname);
+ WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(preserve);
+}
+
static void
_outTypeName(StringInfo str, const TypeName *node)
{
@@ -4258,6 +4268,9 @@ outNode(StringInfo str, const void *obj)
case T_ColumnDef:
_outColumnDef(str, obj);
break;
+ case T_ColumnCompression:
+ _outColumnCompression(str, obj);
+ break;
case T_TypeName:
_outTypeName(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 30acfe615d..9eb2b04d58 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -596,7 +596,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> hash_partbound
%type <defelt> hash_partbound_elem
-%type <str> optColumnCompression
+%type <node> optColumnCompression alterColumnCompression
+%type <str> compressionClause
+%type <list> optCompressionPreserve
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
@@ -2309,12 +2311,12 @@ alter_table_cmd:
$$ = (Node *)n;
}
/* ALTER TABLE <name> ALTER [COLUMN] <colname> SET (COMPRESSION <cm>) */
- | ALTER opt_column ColId SET optColumnCompression
+ | ALTER opt_column ColId SET alterColumnCompression
{
AlterTableCmd *n = makeNode(AlterTableCmd);
n->subtype = AT_SetCompression;
n->name = $3;
- n->def = (Node *) makeString($5);
+ n->def = $5;
$$ = (Node *)n;
}
/* ALTER TABLE <name> DROP [COLUMN] IF EXISTS <colname> [RESTRICT|CASCADE] */
@@ -3437,7 +3439,7 @@ columnDef: ColId Typename optColumnCompression create_generic_options ColQualLis
ColumnDef *n = makeNode(ColumnDef);
n->colname = $1;
n->typeName = $2;
- n->compression = $3;
+ n->compression = (ColumnCompression *) $3;
n->inhcount = 0;
n->is_local = true;
n->is_not_null = false;
@@ -3492,13 +3494,43 @@ columnOptions: ColId ColQualList
}
;
+optCompressionPreserve:
+ PRESERVE '(' name_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+compressionClause:
+ COMPRESSION name { $$ = pstrdup($2); }
+ ;
+
optColumnCompression:
- COMPRESSION name
- {
- $$ = $2;
- }
- | /*EMPTY*/ { $$ = NULL; }
- ;
+ compressionClause
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+alterColumnCompression:
+ compressionClause optCompressionPreserve
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve = (List *) $2;
+ $$ = (Node *) n;
+ }
+ | compressionClause PRESERVE ALL
+ {
+ ColumnCompression *n = makeNode(ColumnCompression);
+ n->cmname = $1;
+ n->preserve_all = true;
+ n->preserve = NIL;
+ $$ = (Node *) n;
+ }
+ ;
ColQualList:
ColQualList ColConstraint { $$ = lappend($1, $2); }
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index e262ec5bcb..f50b50cc8e 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = get_am_name(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute->attcompression);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 46044cb92a..7bf345a4ac 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -9037,6 +9037,80 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
}
PQclear(res);
}
+
+ /*
+ * Get compression info
+ */
+ if (fout->remoteVersion >= 140000 && dopt->binary_upgrade)
+ {
+ int i_amname;
+ int i_amoid;
+ int i_curattnum;
+ int start;
+
+ pg_log_info("finding compression info for table \"%s.%s\"",
+ tbinfo->dobj.namespace->dobj.name,
+ tbinfo->dobj.name);
+
+ tbinfo->attcompression = pg_malloc0(tbinfo->numatts * sizeof(AttrCompressionInfo *));
+
+ resetPQExpBuffer(q);
+ appendPQExpBuffer(q,
+ " SELECT attrelid::pg_catalog.regclass AS relname, attname,"
+ " amname, am.oid as amoid, d.objsubid AS curattnum"
+ " FROM pg_depend d"
+ " JOIN pg_attribute a ON"
+ " (classid = 'pg_class'::pg_catalog.regclass::pg_catalog.oid AND a.attrelid = d.objid"
+ " AND a.attnum = d.objsubid AND d.deptype = 'n'"
+ " AND d.refclassid = 'pg_am'::pg_catalog.regclass::pg_catalog.oid)"
+ " JOIN pg_am am ON"
+ " (d.deptype = 'n' AND d.refobjid = am.oid)"
+ " WHERE (deptype = 'n' AND d.objid = %d AND a.attcompression != am.oid);",
+ tbinfo->dobj.catId.oid);
+
+ res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK);
+ ntups = PQntuples(res);
+
+ if (ntups > 0)
+ {
+ int j;
+ int k;
+
+ i_amname = PQfnumber(res, "amname");
+ i_amoid = PQfnumber(res, "amoid");
+ i_curattnum = PQfnumber(res, "curattnum");
+
+ start = 0;
+
+ for (j = 0; j < ntups; j++)
+ {
+ int attnum = atoi(PQgetvalue(res, j, i_curattnum));
+
+ if ((j == ntups - 1) || atoi(PQgetvalue(res, j + 1, i_curattnum)) != attnum)
+ {
+ AttrCompressionInfo *cminfo = pg_malloc(sizeof(AttrCompressionInfo));
+
+ cminfo->nitems = j - start + 1;
+ cminfo->items = pg_malloc(sizeof(AttrCompressionItem *) * cminfo->nitems);
+
+ for (k = start; k < start + cminfo->nitems; k++)
+ {
+ AttrCompressionItem *cmitem = pg_malloc0(sizeof(AttrCompressionItem));
+
+ cmitem->amname = pg_strdup(PQgetvalue(res, k, i_amname));
+ cmitem->amoid = atooid(PQgetvalue(res, k, i_amoid));
+
+ cminfo->items[k - start] = cmitem;
+ }
+
+ tbinfo->attcompression[attnum - 1] = cminfo;
+ start = j + 1; /* start from next */
+ }
+ }
+ }
+
+ PQclear(res);
+ }
}
destroyPQExpBuffer(q);
@@ -16343,6 +16417,33 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
qualrelname,
fmtId(tbinfo->attnames[j]),
tbinfo->attfdwoptions[j]);
+
+ /*
+ * Dump per-column compression options
+ */
+ if (tbinfo->attcompression && tbinfo->attcompression[j])
+ {
+ AttrCompressionInfo *cminfo = tbinfo->attcompression[j];
+
+ appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
+ qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+
+ if (cminfo->nitems > 0)
+ {
+ appendPQExpBuffer(q, "\nPRESERVE (");
+ for (int i = 0; i < cminfo->nitems; i++)
+ {
+ AttrCompressionItem *item = cminfo->items[i];
+
+ if (i == 0)
+ appendPQExpBuffer(q, "%s", item->amname);
+ else
+ appendPQExpBuffer(q, ", %s", item->amname);
+ }
+ appendPQExpBuffer(q, ")");
+ }
+ appendPQExpBuffer(q, ";\n");
+ }
}
if (ftoptions)
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1789e18f46..a829528cd0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,7 +327,8 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
-
+ struct _attrCompressionInfo **attcompression; /* per-attribute all
+ compression data */
/*
* Stuff computed only for dumpable tables.
*/
@@ -356,6 +357,18 @@ typedef struct _attrDefInfo
bool separate; /* true if must dump as separate item */
} AttrDefInfo;
+typedef struct _attrCompressionItem
+{
+ Oid amoid; /* attribute compression oid */
+ char *amname; /* compression access method name */
+} AttrCompressionItem;
+
+typedef struct _attrCompressionInfo
+{
+ int nitems;
+ AttrCompressionItem **items;
+} AttrCompressionInfo;
+
typedef struct _tableDataInfo
{
DumpableObject dobj;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 2aedddc1d8..05c782e8e9 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2112,6 +2112,13 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "("))
COMPLETE_WITH("n_distinct", "n_distinct_inherited");
+ /* ALTER TABLE ALTER [COLUMN] <foo> SET COMPRESSION */
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny) ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny))
+ COMPLETE_WITH("PRESERVE");
+ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE") ||
+ Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION", MatchAny, "PRESERVE"))
+ COMPLETE_WITH("( ", "ALL");
/* ALTER TABLE ALTER [COLUMN] <foo> SET STORAGE */
else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "STORAGE") ||
Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "STORAGE"))
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index e5aea8a240..bd53f9bb0f 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -143,6 +143,13 @@ extern Oid get_compression_am_oid(const char *amname, bool missing_ok);
extern Oid get_am_oid(const char *amname, bool missing_ok);
extern char *get_am_name(Oid amOid);
+/* commands/compressioncmds.c */
+extern Oid GetAttributeCompression(Form_pg_attribute att,
+ ColumnCompression *compression,
+ bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+
/* support routines in commands/define.c */
extern char *defGetString(DefElem *def);
diff --git a/src/include/commands/tablecmds.h b/src/include/commands/tablecmds.h
index b3d30acc35..e6c98e65d4 100644
--- a/src/include/commands/tablecmds.h
+++ b/src/include/commands/tablecmds.h
@@ -97,5 +97,7 @@ extern void RangeVarCallbackOwnsRelation(const RangeVar *relation,
Oid relId, Oid oldRelId, void *arg);
extern bool PartConstraintImpliedByRelConstraint(Relation scanrel,
List *partConstraint);
+extern void add_column_compression_dependency(Oid relid, int32 attnum,
+ Oid cmoid);
#endif /* TABLECMDS_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 20d6f96f62..24deaad253 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -481,6 +481,7 @@ typedef enum NodeTag
T_PartitionBoundSpec,
T_PartitionRangeDatum,
T_PartitionCmd,
+ T_ColumnCompression,
T_VacuumRelation,
/*
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f9a87dee02..ce0913e18a 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -623,6 +623,20 @@ typedef struct RangeTableSample
int location; /* method name location, or -1 if unknown */
} RangeTableSample;
+/*
+ * ColumnCompression - compression parameters for some attribute
+ *
+ * This represents compression information defined using clause:
+ * .. COMPRESSION <compression method> PRESERVE <compression methods>
+ */
+typedef struct ColumnCompression
+{
+ NodeTag type;
+ char *cmname;
+ bool preserve_all;
+ List *preserve;
+} ColumnCompression;
+
/*
* ColumnDef - column definition (used in various creates)
*
@@ -646,7 +660,7 @@ typedef struct ColumnDef
NodeTag type;
char *colname; /* name of column */
TypeName *typeName; /* type of column */
- char *compression; /* compression method for column */
+ ColumnCompression *compression; /* column compression */
int inhcount; /* number of times column is inherited */
bool is_local; /* column has local (non-inherited) def'n */
bool is_not_null; /* NOT NULL constraint specified? */
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 21c1b451d2..3ed33b6534 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -226,12 +226,47 @@ SELECT pg_column_compression(f1) FROM cmpart;
lz4
(2 rows)
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 64c5855bf7..36a5f8ba5e 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -205,12 +205,49 @@ SELECT pg_column_compression(f1) FROM cmpart;
ERROR: relation "cmpart" does not exist
LINE 1: SELECT pg_column_compression(f1) FROM cmpart;
^
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+Indexes:
+ "idx" btree (f1)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+(2 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
-(1 row)
+ 10040
+(2 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..8f984510ac 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2066,19 +2066,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
REINDEX INDEX CONCURRENTLY concur_reindex_ind1;
REINDEX TABLE CONCURRENTLY concur_reindex_tab;
@@ -2095,19 +2097,21 @@ WHERE classid = 'pg_class'::regclass AND
'concur_reindex_ind4'::regclass,
'concur_reindex_matview'::regclass)
ORDER BY 1, 2;
- obj | objref | deptype
-------------------------------------------+------------------------------------------------------------+---------
- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
- index concur_reindex_ind2 | collation "default" | n
- index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
- index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind3 | table concur_reindex_tab | a
- index concur_reindex_ind4 | collation "default" | n
- index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
- index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
- materialized view concur_reindex_matview | schema public | n
- table concur_reindex_tab | schema public | n
-(10 rows)
+ obj | objref | deptype
+-------------------------------------------------------+------------------------------------------------------------+---------
+ column c2 of materialized view concur_reindex_matview | access method pglz | n
+ column c2 of table concur_reindex_tab | access method pglz | n
+ index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i
+ index concur_reindex_ind2 | collation "default" | n
+ index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind3 | table concur_reindex_tab | a
+ index concur_reindex_ind4 | collation "default" | n
+ index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a
+ index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a
+ materialized view concur_reindex_matview | schema public | n
+ table concur_reindex_tab | schema public | n
+(12 rows)
-- Check that comments are preserved
CREATE TABLE testcomment (i int);
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index b9daa33b74..5774b55d82 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -96,6 +96,15 @@ ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz;
ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4;
SELECT pg_column_compression(f1) FROM cmpart;
+-- preserve old compression method
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (lz4);
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
+SELECT pg_column_compression(f1) FROM cmdata;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
[text/x-diff] 0009-Create-custom-compression-methods.patch (35.6K, ../20210218211314.GC14772@telsasoft.com/10-0009-Create-custom-compression-methods.patch)
download | inline diff:
From 54fbe5935051558be3dff4989936cb163147a756 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilipkumar@localhost.localdomain>
Date: Wed, 10 Feb 2021 16:27:04 +0530
Subject: [PATCH 09/12] Create custom compression methods
Provide syntax to create custom compression methods.
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
doc/src/sgml/ref/alter_table.sgml | 6 ++-
doc/src/sgml/ref/create_access_method.sgml | 12 +++--
doc/src/sgml/ref/create_table.sgml | 9 ++--
src/backend/access/common/detoast.c | 51 ++++++++++++++++---
src/backend/access/common/toast_internals.c | 19 +++++--
src/backend/access/compression/compress_lz4.c | 21 ++++----
.../access/compression/compress_pglz.c | 20 ++++----
src/backend/access/index/amapi.c | 51 ++++++++++++++-----
src/backend/commands/amcmds.c | 22 +++++++-
src/backend/parser/gram.y | 1 +
src/backend/utils/adt/pg_upgrade_support.c | 10 ++++
src/bin/pg_dump/pg_dump.c | 8 +++
src/bin/psql/tab-complete.c | 6 +++
src/include/access/amapi.h | 1 +
src/include/access/compressamapi.h | 45 ++++++++++++++--
src/include/access/toast_internals.h | 16 ++++++
src/include/catalog/binary_upgrade.h | 2 +
src/include/catalog/pg_proc.dat | 4 ++
src/include/postgres.h | 8 +++
src/test/regress/expected/compression.out | 40 ++++++++++++++-
src/test/regress/expected/compression_1.out | 41 ++++++++++++++-
src/test/regress/sql/compression.sql | 10 ++++
22 files changed, 344 insertions(+), 59 deletions(-)
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index c9f443a59c..49c43df1c1 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -391,8 +391,10 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</term>
<listitem>
<para>
- This sets the compression method for a column. The supported compression
- methods are <literal>pglz</literal> and <literal>lz4</literal>.
+ This sets the compression method for a column. The Compression method
+ could be created with <xref linkend="sql-create-access-method"/> or
+ it can be set to any available compression method. The supported buit-in
+ compression methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
was used when building <productname>PostgreSQL</productname>.
The <literal>PRESERVE</literal> list contains a list of compression
diff --git a/doc/src/sgml/ref/create_access_method.sgml b/doc/src/sgml/ref/create_access_method.sgml
index dae43dbaed..c5ef8b738d 100644
--- a/doc/src/sgml/ref/create_access_method.sgml
+++ b/doc/src/sgml/ref/create_access_method.sgml
@@ -61,8 +61,8 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
<listitem>
<para>
This clause specifies the type of access method to define.
- Only <literal>TABLE</literal> and <literal>INDEX</literal>
- are supported at present.
+ Currently, <literal>TABLE</literal>, <literal>INDEX</literal> and
+ <literal>COMPRESSION</literal> access methods are supported.
</para>
</listitem>
</varlistentry>
@@ -77,11 +77,13 @@ CREATE ACCESS METHOD <replaceable class="parameter">name</replaceable>
declared to take a single argument of type <type>internal</type>,
and its return type depends on the type of access method;
for <literal>TABLE</literal> access methods, it must
- be <type>table_am_handler</type> and for <literal>INDEX</literal>
- access methods, it must be <type>index_am_handler</type>.
+ be <type>table_am_handler</type>, for <literal>INDEX</literal>
+ access methods, it must be <type>index_am_handler</type> and
+ for <literal>COMPRESSION</literal> access methods, it must be
+ <type>compression_am_handler</type>.
The C-level API that the handler function must implement varies
depending on the type of access method. The table access method API
- is described in <xref linkend="tableam"/> and the index access method
+ is described in <xref linkend="tableam"/>, the index access method
API is described in <xref linkend="indexam"/>.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index 807597e648..5e13b3288c 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -1004,11 +1004,14 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
column storage types.) Setting this property for a partitioned table
has no direct effect, because such tables have no storage of their own,
but the configured value is inherited by newly-created partitions.
- The supported compression methods are <literal>pglz</literal> and
+ The supported built-in compression methods are <literal>pglz</literal> and
<literal>lz4</literal>. <literal>lz4</literal> is available only if
<literal>--with-lz4</literal> was used when building
- <productname>PostgreSQL</productname>. The default
- is <literal>pglz</literal>.
+ <productname>PostgreSQL</productname>.
+ Compression methods can be created with <xref
+ linkend="sql-create-access-method"/> or it can be set to any available
+ compression method.
+ The default is <literal>pglz</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index 95d5b1c12a..fd7c42c95e 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -462,10 +462,17 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset,
*
* Returns the Oid of the compression method stored in the compressed data. If
* the varlena is not compressed then returns InvalidOid.
+ *
+ * For built-in methods, we only store the built-in compression method id in
+ * first 2-bits of the rawsize and that is directly mapped to the compression
+ * method Oid. And, for the custom compression method we store the Oid of the
+ * compression method in the custom compression header.
*/
Oid
toast_get_compression_oid(struct varlena *attr)
{
+ CompressionId cmid;
+
if (VARATT_IS_EXTERNAL_ONDISK(attr))
{
struct varatt_external toast_pointer;
@@ -485,7 +492,21 @@ toast_get_compression_oid(struct varlena *attr)
else if (!VARATT_IS_COMPRESSED(attr))
return InvalidOid;
- return CompressionIdToOid(TOAST_COMPRESS_METHOD(attr));
+ /*
+ * If it is custom compression id then get the Oid from the custom
+ * compression header otherwise, directly translate the built-in
+ * compression id to compression method Oid.
+ */
+ cmid = TOAST_COMPRESS_METHOD(attr);
+ if (IsCustomCompression(cmid))
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ return hdr->cmoid;
+ }
+ else
+ return CompressionIdToOid(cmid);
}
/* ----------
@@ -494,7 +515,7 @@ toast_get_compression_oid(struct varlena *attr)
* helper function for toast_decompress_datum and toast_decompress_datum_slice
*/
static inline const CompressionAmRoutine *
-toast_get_compression_handler(struct varlena *attr)
+toast_get_compression_handler(struct varlena *attr, int32 *header_size)
{
const CompressionAmRoutine *cmroutine;
CompressionId cmid;
@@ -507,10 +528,21 @@ toast_get_compression_handler(struct varlena *attr)
{
case PGLZ_COMPRESSION_ID:
cmroutine = &pglz_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
case LZ4_COMPRESSION_ID:
cmroutine = &lz4_compress_methods;
+ *header_size = TOAST_COMPRESS_HDRSZ;
break;
+ case CUSTOM_COMPRESSION_ID:
+ {
+ toast_compress_header_custom *hdr;
+
+ hdr = (toast_compress_header_custom *) attr;
+ cmroutine = GetCompressionAmRoutineByAmId(hdr->cmoid);
+ *header_size = TOAST_CUSTOM_COMPRESS_HDRSZ;
+ break;
+ }
default:
elog(ERROR, "invalid compression method id %d", cmid);
}
@@ -526,9 +558,11 @@ toast_get_compression_handler(struct varlena *attr)
static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
@@ -542,16 +576,19 @@ toast_decompress_datum(struct varlena *attr)
static struct varlena *
toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
{
- const CompressionAmRoutine *cmroutine = toast_get_compression_handler(attr);
+ int32 header_size;
+ const CompressionAmRoutine *cmroutine =
+ toast_get_compression_handler(attr, &header_size);
/*
* If the handler supports the slice decompression then decompress the
* slice otherwise decompress complete data.
*/
if (cmroutine->datum_decompress_slice)
- return cmroutine->datum_decompress_slice(attr, slicelength);
+ return cmroutine->datum_decompress_slice(attr, header_size,
+ slicelength);
else
- return cmroutine->datum_decompress(attr);
+ return cmroutine->datum_decompress(attr, header_size);
}
/* ----------
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index b04c5a5eb8..a3539065d3 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -48,6 +48,7 @@ toast_compress_datum(Datum value, Oid cmoid)
{
struct varlena *tmp = NULL;
int32 valsize;
+ bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -65,11 +66,16 @@ toast_compress_datum(Datum value, Oid cmoid)
cmroutine = &lz4_compress_methods;
break;
default:
- elog(ERROR, "Invalid compression method oid %u", cmoid);
+ isCustomCompression = true;
+ cmroutine = GetCompressionAmRoutineByAmId(cmoid);
+ break;
}
/* Call the actual compression function */
- tmp = cmroutine->datum_compress((const struct varlena *) value);
+ tmp = cmroutine->datum_compress((const struct varlena *)value,
+ isCustomCompression ?
+ TOAST_CUSTOM_COMPRESS_HDRSZ :
+ TOAST_COMPRESS_HDRSZ);
if (!tmp)
return PointerGetDatum(NULL);
@@ -88,7 +94,14 @@ toast_compress_datum(Datum value, Oid cmoid)
if (VARSIZE(tmp) < valsize - 2)
{
/* successful compression */
- TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, CompressionOidToId(cmoid));
+ TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, isCustomCompression ?
+ CUSTOM_COMPRESSION_ID :
+ CompressionOidToId(cmoid));
+
+ /* For custom compression, set the oid of the compression method */
+ if (isCustomCompression)
+ TOAST_COMPRESS_SET_CMOID(tmp, cmoid);
+
return PointerGetDatum(tmp);
}
else
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 3079eff7eb..2a3c162836 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -27,7 +27,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value)
+lz4_cmcompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -46,10 +46,10 @@ lz4_cmcompress(const struct varlena *value)
* that will be needed for varlena overhead, and allocate that amount.
*/
max_size = LZ4_compressBound(valsize);
- tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS);
+ tmp = (struct varlena *) palloc(max_size + header_size);
len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
valsize, max_size);
if (len <= 0)
elog(ERROR, "could not compress data with lz4");
@@ -61,7 +61,7 @@ lz4_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
#endif
@@ -73,7 +73,7 @@ lz4_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress(const struct varlena *value)
+lz4_cmdecompress(const struct varlena *value, int32 header_size)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -87,9 +87,9 @@ lz4_cmdecompress(const struct varlena *value)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress the data */
- rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
ereport(ERROR,
@@ -109,7 +109,8 @@ lz4_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
+lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
+ int32 slicelength)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -123,9 +124,9 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
/* decompress partial data using lz4 routine */
- rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
+ rawsize = LZ4_decompress_safe_partial((char *) value + header_size,
VARDATA(result),
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ VARSIZE(value) - header_size,
slicelength,
VARRAWSIZE_4B_C(value));
if (rawsize < 0)
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 8a4bf427cf..7f6e7429fe 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -26,7 +26,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value)
+pglz_cmcompress(const struct varlena *value, int32 header_size)
{
int32 valsize,
len;
@@ -47,11 +47,11 @@ pglz_cmcompress(const struct varlena *value)
* and allocate the memory for holding the compressed data and the header.
*/
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- VARHDRSZ_COMPRESS);
+ header_size);
len = pglz_compress(VARDATA_ANY(value),
valsize,
- (char *) tmp + VARHDRSZ_COMPRESS,
+ (char *) tmp + header_size,
NULL);
if (len < 0)
{
@@ -59,7 +59,7 @@ pglz_cmcompress(const struct varlena *value)
return NULL;
}
- SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS);
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
return tmp;
}
@@ -70,15 +70,15 @@ pglz_cmcompress(const struct varlena *value)
* Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress(const struct varlena *value)
+pglz_cmdecompress(const struct varlena *value, int32 header_size)
{
struct varlena *result;
int32 rawsize;
result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
VARRAWSIZE_4B_C(value), true);
if (rawsize < 0)
@@ -97,7 +97,7 @@ pglz_cmdecompress(const struct varlena *value)
* Decompresses part of the data. Returns the decompressed varlena.
*/
static struct varlena *
-pglz_cmdecompress_slice(const struct varlena *value,
+pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
int32 slicelength)
{
struct varlena *result;
@@ -105,8 +105,8 @@ pglz_cmdecompress_slice(const struct varlena *value,
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS,
- VARSIZE(value) - VARHDRSZ_COMPRESS,
+ rawsize = pglz_decompress((char *) value + header_size,
+ VARSIZE(value) - header_size,
VARDATA(result),
slicelength, false);
if (rawsize < 0)
diff --git a/src/backend/access/index/amapi.c b/src/backend/access/index/amapi.c
index d30bc43514..16f8fab359 100644
--- a/src/backend/access/index/amapi.c
+++ b/src/backend/access/index/amapi.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/amapi.h"
+#include "access/compressamapi.h"
#include "access/htup_details.h"
#include "catalog/pg_am.h"
#include "catalog/pg_opclass.h"
@@ -46,14 +47,14 @@ GetIndexAmRoutine(Oid amhandler)
}
/*
- * GetIndexAmRoutineByAmId - look up the handler of the index access method
- * with the given OID, and get its IndexAmRoutine struct.
+ * GetAmHandlerByAmId - look up the handler of the index/compression access
+ * method with the given OID, and get its handler function.
*
- * If the given OID isn't a valid index access method, returns NULL if
- * noerror is true, else throws error.
+ * If the given OID isn't a valid index/compression access method, returns
+ * Invalid Oid if noerror is true, else throws error.
*/
-IndexAmRoutine *
-GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+regproc
+GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror)
{
HeapTuple tuple;
Form_pg_am amform;
@@ -64,24 +65,25 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (!HeapTupleIsValid(tuple))
{
if (noerror)
- return NULL;
+ return InvalidOid;
elog(ERROR, "cache lookup failed for access method %u",
amoid);
}
amform = (Form_pg_am) GETSTRUCT(tuple);
/* Check if it's an index access method as opposed to some other AM */
- if (amform->amtype != AMTYPE_INDEX)
+ if (amform->amtype != amtype)
{
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("access method \"%s\" is not of type %s",
- NameStr(amform->amname), "INDEX")));
+ NameStr(amform->amname), (amtype == AMTYPE_INDEX ?
+ "INDEX" : "COMPRESSION"))));
}
amhandler = amform->amhandler;
@@ -92,16 +94,41 @@ GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
if (noerror)
{
ReleaseSysCache(tuple);
- return NULL;
+ return InvalidOid;
}
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("index access method \"%s\" does not have a handler",
+ errmsg("access method \"%s\" does not have a handler",
NameStr(amform->amname))));
}
ReleaseSysCache(tuple);
+ return amhandler;
+}
+
+/*
+ * GetIndexAmRoutineByAmId - look up the handler of the index access method
+ * with the given OID, and get its IndexAmRoutine struct.
+ *
+ * If the given OID isn't a valid index access method, returns NULL if
+ * noerror is true, else throws error.
+ */
+IndexAmRoutine *
+GetIndexAmRoutineByAmId(Oid amoid, bool noerror)
+{
+ regproc amhandler;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_INDEX, noerror);
+
+ /* Complain if handler OID is invalid */
+ if (!OidIsValid(amhandler))
+ {
+ Assert(noerror);
+ return NULL;
+ }
+
/* And finally, call the handler function to get the API struct. */
return GetIndexAmRoutine(amhandler);
}
diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c
index 1682afd2a4..0ac86eee02 100644
--- a/src/backend/commands/amcmds.c
+++ b/src/backend/commands/amcmds.c
@@ -45,6 +45,9 @@ static Oid default_toast_compression_oid = InvalidOid;
static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* Set by pg_upgrade_support functions */
+Oid binary_upgrade_next_pg_am_oid = InvalidOid;
+
/*
* CreateAccessMethod
* Registers a new access method.
@@ -93,7 +96,19 @@ CreateAccessMethod(CreateAmStmt *stmt)
memset(values, 0, sizeof(values));
memset(nulls, false, sizeof(nulls));
- amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+ if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_pg_am_oid))
+ {
+ /* amoid should be found in some cases */
+ if (binary_upgrade_next_pg_am_oid < FirstNormalObjectId &&
+ (!OidIsValid(amoid) || binary_upgrade_next_pg_am_oid != amoid))
+ elog(ERROR, "could not link to built-in attribute compression");
+
+ amoid = binary_upgrade_next_pg_am_oid;
+ binary_upgrade_next_pg_am_oid = InvalidOid;
+ }
+ else
+ amoid = GetNewOidWithIndex(rel, AmOidIndexId, Anum_pg_am_oid);
+
values[Anum_pg_am_oid - 1] = ObjectIdGetDatum(amoid);
values[Anum_pg_am_amname - 1] =
DirectFunctionCall1(namein, CStringGetDatum(stmt->amname));
@@ -237,6 +252,8 @@ get_am_type_string(char amtype)
return "INDEX";
case AMTYPE_TABLE:
return "TABLE";
+ case AMTYPE_COMPRESSION:
+ return "COMPRESSION";
default:
/* shouldn't happen */
elog(ERROR, "invalid access method type '%c'", amtype);
@@ -274,6 +291,9 @@ lookup_am_handler_func(List *handler_name, char amtype)
case AMTYPE_TABLE:
expectedType = TABLE_AM_HANDLEROID;
break;
+ case AMTYPE_COMPRESSION:
+ expectedType = COMPRESSION_AM_HANDLEROID;
+ break;
default:
elog(ERROR, "unrecognized access method type \"%c\"", amtype);
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 9eb2b04d58..b22ce818cd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -5305,6 +5305,7 @@ CreateAmStmt: CREATE ACCESS METHOD name TYPE_P am_type HANDLER handler_name
am_type:
INDEX { $$ = AMTYPE_INDEX; }
| TABLE { $$ = AMTYPE_TABLE; }
+ | COMPRESSION { $$ = AMTYPE_COMPRESSION; }
;
/*****************************************************************************
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index a575c95079..f026104c01 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -128,6 +128,16 @@ binary_upgrade_set_next_pg_authid_oid(PG_FUNCTION_ARGS)
PG_RETURN_VOID();
}
+Datum
+binary_upgrade_set_next_pg_am_oid(PG_FUNCTION_ARGS)
+{
+ Oid amoid = PG_GETARG_OID(0);
+
+ CHECK_IS_BINARY_UPGRADE;
+ binary_upgrade_next_pg_am_oid = amoid;
+ PG_RETURN_VOID();
+}
+
Datum
binary_upgrade_create_empty_extension(PG_FUNCTION_ARGS)
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 7bf345a4ac..88fa7e1ed3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -13136,6 +13136,11 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
qamname = pg_strdup(fmtId(aminfo->dobj.name));
+ if (dopt->binary_upgrade && aminfo->amtype == AMTYPE_COMPRESSION)
+ appendPQExpBuffer(q,
+ "SELECT pg_catalog.binary_upgrade_set_next_pg_am_oid('%u'::pg_catalog.oid);\n",
+ aminfo->dobj.catId.oid);
+
appendPQExpBuffer(q, "CREATE ACCESS METHOD %s ", qamname);
switch (aminfo->amtype)
@@ -13146,6 +13151,9 @@ dumpAccessMethod(Archive *fout, const AccessMethodInfo *aminfo)
case AMTYPE_TABLE:
appendPQExpBufferStr(q, "TYPE TABLE ");
break;
+ case AMTYPE_COMPRESSION:
+ appendPQExpBufferStr(q, "TYPE COMPRESSION ");
+ break;
default:
pg_log_warning("invalid type \"%c\" of access method \"%s\"",
aminfo->amtype, qamname);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 05c782e8e9..fe4be7bda4 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -883,6 +883,12 @@ static const SchemaQuery Query_for_list_of_statistics = {
" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
" amtype=" CppAsString2(AMTYPE_TABLE)
+#define Query_for_list_of_compression_access_methods \
+" SELECT pg_catalog.quote_ident(amname) "\
+" FROM pg_catalog.pg_am "\
+" WHERE substring(pg_catalog.quote_ident(amname),1,%d)='%s' AND "\
+" amtype=" CppAsString2(AMTYPE_COMPRESSION)
+
/* the silly-looking length condition is just to eat up the current word */
#define Query_for_list_of_arguments \
"SELECT pg_catalog.oidvectortypes(proargtypes)||')' "\
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 1513cafcf4..bc1e2437f0 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -285,6 +285,7 @@ typedef struct IndexAmRoutine
/* Functions in access/index/amapi.c */
extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler);
+extern regproc GetAmHandlerByAmId(Oid amoid, char amtype, bool noerror);
extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror);
void InitializeAccessMethods(void);
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index d75a8e9df2..bac7b47cfc 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -15,10 +15,13 @@
#include "postgres.h"
+#include "fmgr.h"
+#include "access/amapi.h"
#include "catalog/pg_am_d.h"
#include "nodes/nodes.h"
#include "utils/guc.h"
+
/*
* Built-in compression method-id. The toast compression header will store
* this in the first 2 bits of the raw length. These built-in compression
@@ -27,7 +30,9 @@
typedef enum CompressionId
{
PGLZ_COMPRESSION_ID = 0,
- LZ4_COMPRESSION_ID = 1
+ LZ4_COMPRESSION_ID = 1,
+ /* one free slot for the future built-in method */
+ CUSTOM_COMPRESSION_ID = 3
} CompressionId;
/* Default compression method if not specified. */
@@ -41,13 +46,19 @@ extern bool check_default_toast_compression(char **newval, void **extra, GucSour
extern Oid GetDefaultToastCompression(void);
+#define IsCustomCompression(cmid) ((cmid) == CUSTOM_COMPRESSION_ID)
+
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
-typedef struct varlena *(*cmcompress_function) (const struct varlena *value);
-typedef struct varlena *(*cmdecompress_function) (const struct varlena *value);
+typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
+ int32 toast_header_size);
+typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
+ int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
- (const struct varlena *value, int32 slicelength);
+ (const struct varlena *value,
+ int32 toast_header_size,
+ int32 slicelength);
/*
* API struct for a compression AM.
@@ -106,4 +117,30 @@ CompressionIdToOid(CompressionId cmid)
}
}
+/*
+ * GetCompressionAmRoutineByAmId - look up the handler of the compression access
+ * method with the given OID, and get its CompressionAmRoutine struct.
+ */
+static inline CompressionAmRoutine *
+GetCompressionAmRoutineByAmId(Oid amoid)
+{
+ regproc amhandler;
+ Datum datum;
+ CompressionAmRoutine *routine;
+
+ /* Get handler function OID for the access method */
+ amhandler = GetAmHandlerByAmId(amoid, AMTYPE_COMPRESSION, false);
+ Assert(OidIsValid(amhandler));
+
+ /* And finally, call the handler function to get the API struct */
+ datum = OidFunctionCall0(amhandler);
+ routine = (CompressionAmRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, CompressionAmRoutine))
+ elog(ERROR, "compression access method handler function %u did not return an CompressionAmRoutine struct",
+ amhandler);
+
+ return routine;
+}
+
#endif /* COMPRESSAMAPI_H */
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index 31ff91a09c..ac28f9ed55 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -27,10 +27,23 @@ typedef struct toast_compress_header
* rawsize */
} toast_compress_header;
+/*
+ * If the compression method were used, then data also contains
+ * Oid of compression options
+ */
+typedef struct toast_compress_header_custom
+{
+ int32 vl_len_; /* varlena header (do not touch directly!) */
+ uint32 info; /* 2 bits for compression method + rawsize */
+ Oid cmoid; /* Oid from pg_am */
+} toast_compress_header_custom;
+
/*
* Utilities for manipulation of header information for compressed
* toast entries.
*/
+#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
+#define TOAST_CUSTOM_COMPRESS_HDRSZ ((int32)sizeof(toast_compress_header_custom))
#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->info >> VARLENA_RAWSIZE_BITS)
#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \
do { \
@@ -38,6 +51,9 @@ typedef struct toast_compress_header
((toast_compress_header *) (ptr))->info = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \
} while (0)
+#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
+ (((toast_compress_header_custom *)(ptr))->cmoid = (oid))
+
extern Datum toast_compress_datum(Datum value, Oid cmoid);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h
index f6e82e7ac5..9d65bae238 100644
--- a/src/include/catalog/binary_upgrade.h
+++ b/src/include/catalog/binary_upgrade.h
@@ -26,6 +26,8 @@ extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid;
extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid;
+extern PGDLLIMPORT Oid binary_upgrade_next_pg_am_oid;
+
extern PGDLLIMPORT bool binary_upgrade_record_init_privs;
#endif /* BINARY_UPGRADE_H */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4b3d34ae1a..6556fe847e 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10777,6 +10777,10 @@
proname => 'binary_upgrade_set_next_pg_authid_oid', provolatile => 'v',
proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
prosrc => 'binary_upgrade_set_next_pg_authid_oid' },
+{ oid => '2137', descr => 'for use by pg_upgrade',
+ proname => 'binary_upgrade_set_next_pg_am_oid', provolatile => 'v',
+ proparallel => 'r', prorettype => 'void', proargtypes => 'oid',
+ prosrc => 'binary_upgrade_set_next_pg_am_oid' },
{ oid => '3591', descr => 'for use by pg_upgrade',
proname => 'binary_upgrade_create_empty_extension', proisstrict => 'f',
provolatile => 'v', proparallel => 'u', prorettype => 'void',
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 667927fd7c..ede3b11ef3 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -149,6 +149,14 @@ typedef union
* flags */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
+ struct /* Compressed-in-line format */
+ {
+ uint32 va_header;
+ uint32 va_info; /* Original data size (excludes header) and
+ * flags */
+ Oid va_cmid; /* Oid of compression method */
+ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
+ } va_custom_compressed;
} varattrib_4b;
typedef struct
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index 3ed33b6534..e52e272d39 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -260,13 +260,51 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | lz4 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index 36a5f8ba5e..d4e215c931 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -241,13 +241,52 @@ SELECT pg_column_compression(f1) FROM cmdata;
pglz
(2 rows)
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+ pglz2
+(3 rows)
+
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+ERROR: not built with lz4 support
+SELECT pg_column_compression(f1) FROM cmdata;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+ pglz2
+(3 rows)
+
+\d+ cmdata
+ Table "public.cmdata"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz2 | |
+Indexes:
+ "idx" btree (f1)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
--------
10000
10040
-(2 rows)
+ 10040
+(3 rows)
SELECT length(f1) FROM cmdata1;
ERROR: relation "cmdata1" does not exist
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 5774b55d82..9d2e72b784 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -105,6 +105,16 @@ SELECT pg_column_compression(f1) FROM cmdata;
ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE ALL;
SELECT pg_column_compression(f1) FROM cmdata;
+-- create compression method
+CREATE ACCESS METHOD pglz2 TYPE COMPRESSION HANDLER pglzhandler;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION pglz2 PRESERVE ALL;
+INSERT INTO cmdata VALUES (repeat('1234567890',1004));
+\d+ cmdata
+SELECT pg_column_compression(f1) FROM cmdata;
+ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
+SELECT pg_column_compression(f1) FROM cmdata;
+\d+ cmdata
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
[text/x-diff] 0010-new-compression-method-extension-for-zlib.patch (10.0K, ../20210218211314.GC14772@telsasoft.com/11-0010-new-compression-method-extension-for-zlib.patch)
download | inline diff:
From 43fd97de3bc880cff9f53c517be6f99df3e781b7 Mon Sep 17 00:00:00 2001
From: dilipkumar <dilipbalaut@gmail.com>
Date: Fri, 16 Oct 2020 19:56:14 +0530
Subject: [PATCH 10/12] new compression method extension for zlib
Dilip Kumar
---
contrib/Makefile | 1 +
contrib/cmzlib/.gitignore | 4 +
contrib/cmzlib/Makefile | 26 +++++
contrib/cmzlib/cmzlib--1.0.sql | 13 +++
contrib/cmzlib/cmzlib.c | 157 +++++++++++++++++++++++++++++
contrib/cmzlib/cmzlib.control | 5 +
contrib/cmzlib/expected/cmzlib.out | 53 ++++++++++
contrib/cmzlib/sql/cmzlib.sql | 22 ++++
8 files changed, 281 insertions(+)
create mode 100644 contrib/cmzlib/.gitignore
create mode 100644 contrib/cmzlib/Makefile
create mode 100644 contrib/cmzlib/cmzlib--1.0.sql
create mode 100644 contrib/cmzlib/cmzlib.c
create mode 100644 contrib/cmzlib/cmzlib.control
create mode 100644 contrib/cmzlib/expected/cmzlib.out
create mode 100644 contrib/cmzlib/sql/cmzlib.sql
diff --git a/contrib/Makefile b/contrib/Makefile
index f27e458482..9e452d8dd0 100644
--- a/contrib/Makefile
+++ b/contrib/Makefile
@@ -12,6 +12,7 @@ SUBDIRS = \
bloom \
btree_gin \
btree_gist \
+ cmzlib \
citext \
cube \
dblink \
diff --git a/contrib/cmzlib/.gitignore b/contrib/cmzlib/.gitignore
new file mode 100644
index 0000000000..5dcb3ff972
--- /dev/null
+++ b/contrib/cmzlib/.gitignore
@@ -0,0 +1,4 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
diff --git a/contrib/cmzlib/Makefile b/contrib/cmzlib/Makefile
new file mode 100644
index 0000000000..956fbe7cc8
--- /dev/null
+++ b/contrib/cmzlib/Makefile
@@ -0,0 +1,26 @@
+# contrib/cmzlib/Makefile
+
+MODULE_big = cmzlib
+OBJS = \
+ $(WIN32RES) \
+ cmzlib.o
+
+EXTENSION = cmzlib
+DATA = cmzlib--1.0.sql
+PGFILEDESC = "zlib compression method "
+
+SHLIB_LINK += $(filter -lz, $(LIBS))
+
+REGRESS = cmzlib
+
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = contrib/cmzlib
+top_builddir = ../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/contrib/cmzlib/cmzlib--1.0.sql b/contrib/cmzlib/cmzlib--1.0.sql
new file mode 100644
index 0000000000..41f2f95870
--- /dev/null
+++ b/contrib/cmzlib/cmzlib--1.0.sql
@@ -0,0 +1,13 @@
+/* contrib/cm_lz4/cmzlib--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION cmzlib" to load this file. \quit
+
+CREATE FUNCTION zlibhandler(internal)
+RETURNS compression_am_handler
+AS 'MODULE_PATHNAME'
+LANGUAGE C;
+
+-- Compression method
+CREATE ACCESS METHOD zlib TYPE COMPRESSION HANDLER zlibhandler;
+COMMENT ON ACCESS METHOD zlib IS 'zlib compression method';
diff --git a/contrib/cmzlib/cmzlib.c b/contrib/cmzlib/cmzlib.c
new file mode 100644
index 0000000000..686a7c7e0d
--- /dev/null
+++ b/contrib/cmzlib/cmzlib.c
@@ -0,0 +1,157 @@
+/*-------------------------------------------------------------------------
+ *
+ * cmzlib.c
+ * zlib compression method
+ *
+ * Copyright (c) 2015-2018, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * contrib/cmzlib/cmzlib.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+#include "access/compressamapi.h"
+#include "access/toast_internals.h"
+
+#include "fmgr.h"
+#include "utils/builtins.h"
+
+#include <zlib.h>
+
+PG_MODULE_MAGIC;
+
+PG_FUNCTION_INFO_V1(zlibhandler);
+
+void _PG_init(void);
+
+/*
+ * Module initialize function: initialize info about zlib
+ */
+void
+_PG_init(void)
+{
+
+}
+
+#define ZLIB_MAX_DICTIONARY_LENGTH 32768
+#define ZLIB_DICTIONARY_DELIM (" ,")
+
+typedef struct
+{
+ int level;
+ Bytef dict[ZLIB_MAX_DICTIONARY_LENGTH];
+ unsigned int dictlen;
+} zlib_state;
+
+/*
+ * zlib_cmcompress - compression routine for zlib compression method
+ *
+ * Compresses source into dest using the default compression level.
+ * Returns the compressed varlena, or NULL if compression fails.
+ */
+static struct varlena *
+zlib_cmcompress(const struct varlena *value, int32 header_size)
+{
+ int32 valsize,
+ len;
+ struct varlena *tmp = NULL;
+ z_streamp zp;
+ int res;
+ zlib_state state;
+
+ state.level = Z_DEFAULT_COMPRESSION;
+
+ zp = (z_streamp) palloc(sizeof(z_stream));
+ zp->zalloc = Z_NULL;
+ zp->zfree = Z_NULL;
+ zp->opaque = Z_NULL;
+
+ if (deflateInit(zp, state.level) != Z_OK)
+ elog(ERROR, "could not initialize compression library: %s", zp->msg);
+
+ valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value));
+ tmp = (struct varlena *) palloc(valsize + header_size);
+ zp->next_in = (void *) VARDATA_ANY(value);
+ zp->avail_in = valsize;
+ zp->avail_out = valsize;
+ zp->next_out = (void *) ((char *) tmp + header_size);
+
+ do
+ {
+ res = deflate(zp, Z_FINISH);
+ if (res == Z_STREAM_ERROR)
+ elog(ERROR, "could not compress data: %s", zp->msg);
+ } while (zp->avail_in != 0);
+
+ Assert(res == Z_STREAM_END);
+
+ len = valsize - zp->avail_out;
+ if (deflateEnd(zp) != Z_OK)
+ elog(ERROR, "could not close compression stream: %s", zp->msg);
+ pfree(zp);
+
+ if (len > 0)
+ {
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
+ return tmp;
+ }
+
+ pfree(tmp);
+ return NULL;
+}
+
+/*
+ * zlib_cmdecompress - decompression routine for zlib compression method
+ *
+ * Returns the decompressed varlena.
+ */
+static struct varlena *
+zlib_cmdecompress(const struct varlena *value, int32 header_size)
+{
+ struct varlena *result;
+ z_streamp zp;
+ int res = Z_OK;
+
+ zp = (z_streamp) palloc(sizeof(z_stream));
+ zp->zalloc = Z_NULL;
+ zp->zfree = Z_NULL;
+ zp->opaque = Z_NULL;
+
+ if (inflateInit(zp) != Z_OK)
+ elog(ERROR, "could not initialize compression library: %s", zp->msg);
+
+ zp->next_in = (void *) ((char *) value + header_size);
+ zp->avail_in = VARSIZE(value) - header_size;
+ zp->avail_out = VARRAWSIZE_4B_C(value);
+
+ result = (struct varlena *) palloc(zp->avail_out + VARHDRSZ);
+ SET_VARSIZE(result, zp->avail_out + VARHDRSZ);
+ zp->next_out = (void *) VARDATA(result);
+
+ while (zp->avail_in > 0)
+ {
+ res = inflate(zp, 0);
+ if (!(res == Z_OK || res == Z_STREAM_END))
+ elog(ERROR, "could not uncompress data: %s", zp->msg);
+ }
+
+ if (inflateEnd(zp) != Z_OK)
+ elog(ERROR, "could not close compression library: %s", zp->msg);
+
+ pfree(zp);
+ return result;
+}
+
+const CompressionAmRoutine zlib_compress_methods = {
+ .type = T_CompressionAmRoutine,
+ .datum_compress = zlib_cmcompress,
+ .datum_decompress = zlib_cmdecompress,
+ .datum_decompress_slice = NULL};
+
+Datum
+zlibhandler(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_POINTER(&zlib_compress_methods);
+}
diff --git a/contrib/cmzlib/cmzlib.control b/contrib/cmzlib/cmzlib.control
new file mode 100644
index 0000000000..2eb10f3a83
--- /dev/null
+++ b/contrib/cmzlib/cmzlib.control
@@ -0,0 +1,5 @@
+# cm_lz4 extension
+comment = 'cmzlib compression method'
+default_version = '1.0'
+module_pathname = '$libdir/cmzlib'
+relocatable = true
diff --git a/contrib/cmzlib/expected/cmzlib.out b/contrib/cmzlib/expected/cmzlib.out
new file mode 100644
index 0000000000..2b6fac7e0b
--- /dev/null
+++ b/contrib/cmzlib/expected/cmzlib.out
@@ -0,0 +1,53 @@
+CREATE EXTENSION cmzlib;
+-- zlib compression
+CREATE TABLE zlibtest(f1 TEXT COMPRESSION pglz);
+INSERT INTO zlibtest VALUES(repeat('1234567890',1004));
+INSERT INTO zlibtest VALUES(repeat('1234567890 one two three',1004));
+SELECT length(f1) FROM zlibtest;
+ length
+--------
+ 10040
+ 24096
+(2 rows)
+
+-- alter compression method with rewrite
+ALTER TABLE zlibtest ALTER COLUMN f1 SET COMPRESSION pglz;
+\d+ zlibtest
+ Table "public.zlibtest"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+
+ALTER TABLE zlibtest ALTER COLUMN f1 SET COMPRESSION zlib;
+\d+ zlibtest
+ Table "public.zlibtest"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | zlib | |
+
+-- preserve old compression method
+ALTER TABLE zlibtest ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (zlib);
+INSERT INTO zlibtest VALUES (repeat('1234567890',1004));
+\d+ zlibtest
+ Table "public.zlibtest"
+ Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
+--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
+ f1 | text | | | | extended | pglz | |
+
+SELECT pg_column_compression(f1) FROM zlibtest;
+ pg_column_compression
+-----------------------
+ zlib
+ zlib
+ pglz
+(3 rows)
+
+SELECT length(f1) FROM zlibtest;
+ length
+--------
+ 10040
+ 24096
+ 10040
+(3 rows)
+
+DROP TABLE zlibtest;
diff --git a/contrib/cmzlib/sql/cmzlib.sql b/contrib/cmzlib/sql/cmzlib.sql
new file mode 100644
index 0000000000..ea8d206625
--- /dev/null
+++ b/contrib/cmzlib/sql/cmzlib.sql
@@ -0,0 +1,22 @@
+CREATE EXTENSION cmzlib;
+
+-- zlib compression
+CREATE TABLE zlibtest(f1 TEXT COMPRESSION pglz);
+INSERT INTO zlibtest VALUES(repeat('1234567890',1004));
+INSERT INTO zlibtest VALUES(repeat('1234567890 one two three',1004));
+SELECT length(f1) FROM zlibtest;
+
+-- alter compression method with rewrite
+ALTER TABLE zlibtest ALTER COLUMN f1 SET COMPRESSION pglz;
+\d+ zlibtest
+ALTER TABLE zlibtest ALTER COLUMN f1 SET COMPRESSION zlib;
+\d+ zlibtest
+
+-- preserve old compression method
+ALTER TABLE zlibtest ALTER COLUMN f1 SET COMPRESSION pglz PRESERVE (zlib);
+INSERT INTO zlibtest VALUES (repeat('1234567890',1004));
+\d+ zlibtest
+SELECT pg_column_compression(f1) FROM zlibtest;
+SELECT length(f1) FROM zlibtest;
+
+DROP TABLE zlibtest;
--
2.17.0
[text/x-diff] 0011-Support-compression-methods-options.patch (64.6K, ../20210218211314.GC14772@telsasoft.com/12-0011-Support-compression-methods-options.patch)
download | inline diff:
From cefc9e9428e6aecf58ccfefe31342b6bf97afa8f Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilipkumar@localhost.localdomain>
Date: Wed, 10 Feb 2021 16:29:55 +0530
Subject: [PATCH 11/12] Support compression methods options
Dilip Kumar based on the patches from Ildus Kurbangaliev.
Design input from Tomas Vondra and Robert Haas
Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby
---
contrib/cmzlib/cmzlib.c | 75 +++++++++--
doc/src/sgml/ref/alter_table.sgml | 6 +-
doc/src/sgml/ref/create_table.sgml | 6 +-
src/backend/access/brin/brin_tuple.c | 5 +-
src/backend/access/common/indextuple.c | 4 +-
src/backend/access/common/reloptions.c | 64 ++++++++++
src/backend/access/common/toast_internals.c | 14 ++-
src/backend/access/compression/compress_lz4.c | 75 ++++++++++-
.../access/compression/compress_pglz.c | 116 +++++++++++++++---
src/backend/access/table/toast_helper.c | 8 +-
src/backend/bootstrap/bootparse.y | 1 +
src/backend/catalog/heap.c | 15 ++-
src/backend/catalog/index.c | 43 +++++--
src/backend/catalog/toasting.c | 1 +
src/backend/commands/cluster.c | 1 +
src/backend/commands/compressioncmds.c | 79 +++++++++++-
src/backend/commands/foreigncmds.c | 44 -------
src/backend/commands/tablecmds.c | 102 ++++++++++-----
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/nodes/outfuncs.c | 1 +
src/backend/parser/gram.y | 16 ++-
src/backend/parser/parse_utilcmd.c | 2 +-
src/bin/pg_dump/pg_dump.c | 23 +++-
src/bin/pg_dump/pg_dump.h | 1 +
src/include/access/compressamapi.h | 18 ++-
src/include/access/toast_helper.h | 2 +
src/include/access/toast_internals.h | 2 +-
src/include/catalog/heap.h | 2 +
src/include/catalog/pg_attribute.h | 3 +
src/include/commands/defrem.h | 9 +-
src/include/nodes/parsenodes.h | 1 +
src/test/regress/expected/compression.out | 52 ++++++++
src/test/regress/expected/compression_1.out | 54 ++++++++
src/test/regress/expected/misc_sanity.out | 3 +-
src/test/regress/sql/compression.sql | 18 +++
36 files changed, 727 insertions(+), 141 deletions(-)
diff --git a/contrib/cmzlib/cmzlib.c b/contrib/cmzlib/cmzlib.c
index 686a7c7e0d..d8c2865f21 100644
--- a/contrib/cmzlib/cmzlib.c
+++ b/contrib/cmzlib/cmzlib.c
@@ -14,6 +14,7 @@
#include "postgres.h"
#include "access/compressamapi.h"
#include "access/toast_internals.h"
+#include "commands/defrem.h"
#include "fmgr.h"
#include "utils/builtins.h"
@@ -45,6 +46,62 @@ typedef struct
unsigned int dictlen;
} zlib_state;
+/*
+ * Check options if specified. All validation is located here so
+ * we don't need to do it again in cminitstate function.
+ */
+static void
+zlib_cmcheck(List *options)
+{
+ ListCell *lc;
+
+ foreach(lc, options)
+ {
+ DefElem *def = (DefElem *) lfirst(lc);
+
+ if (strcmp(def->defname, "level") == 0)
+ {
+ int8 level = pg_atoi(defGetString(def), sizeof(int8), 0);
+
+ if (level < 0 || level > 9)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("unexpected value for zlib compression level: \"%s\"",
+ defGetString(def)),
+ errhint("expected value between 0 and 9")
+ ));
+ }
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_PARAMETER),
+ errmsg("unknown compression option for zlib: \"%s\"", def->defname)));
+ }
+}
+
+static void *
+zlib_cminitstate(List *options)
+{
+ zlib_state *state = NULL;
+
+ state = palloc0(sizeof(zlib_state));
+ state->level = Z_DEFAULT_COMPRESSION;
+
+ if (list_length(options) > 0)
+ {
+ ListCell *lc;
+
+ foreach(lc, options)
+ {
+ DefElem *def = (DefElem *) lfirst(lc);
+
+ if (strcmp(def->defname, "level") == 0)
+ state->level = pg_atoi(defGetString(def), sizeof(int), 0);
+ }
+ }
+
+ return state;
+}
+
/*
* zlib_cmcompress - compression routine for zlib compression method
*
@@ -52,23 +109,21 @@ typedef struct
* Returns the compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-zlib_cmcompress(const struct varlena *value, int32 header_size)
+zlib_cmcompress(const struct varlena *value, int32 header_size, void *options)
{
- int32 valsize,
- len;
+ int32 valsize,
+ len;
struct varlena *tmp = NULL;
- z_streamp zp;
- int res;
- zlib_state state;
-
- state.level = Z_DEFAULT_COMPRESSION;
+ z_streamp zp;
+ int res;
+ zlib_state *state = (zlib_state *) options;
zp = (z_streamp) palloc(sizeof(z_stream));
zp->zalloc = Z_NULL;
zp->zfree = Z_NULL;
zp->opaque = Z_NULL;
- if (deflateInit(zp, state.level) != Z_OK)
+ if (deflateInit(zp, state->level) != Z_OK)
elog(ERROR, "could not initialize compression library: %s", zp->msg);
valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value));
@@ -146,6 +201,8 @@ zlib_cmdecompress(const struct varlena *value, int32 header_size)
const CompressionAmRoutine zlib_compress_methods = {
.type = T_CompressionAmRoutine,
+ .datum_check = zlib_cmcheck,
+ .datum_initstate = zlib_cminitstate,
.datum_compress = zlib_cmcompress,
.datum_decompress = zlib_cmdecompress,
.datum_decompress_slice = NULL};
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 49c43df1c1..0f861e78f9 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -54,7 +54,7 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable>
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET ( <replaceable class="parameter">attribute_option</replaceable> = <replaceable class="parameter">value</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> RESET ( <replaceable class="parameter">attribute_option</replaceable> [, ... ] )
ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
- ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
+ ALTER [ COLUMN ] <replaceable class="parameter">column_name</replaceable> SET COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ WITH (<replaceable class="parameter">compression_method_options</replaceable>) ] [ PRESERVE (<replaceable class="parameter">compression_preserve_list</replaceable>) | PRESERVE ALL ]
ADD <replaceable class="parameter">table_constraint</replaceable> [ NOT VALID ]
ADD <replaceable class="parameter">table_constraint_using_index</replaceable>
ALTER CONSTRAINT <replaceable class="parameter">constraint_name</replaceable> [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -396,7 +396,9 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
it can be set to any available compression method. The supported buit-in
compression methods are <literal>pglz</literal> and <literal>lz4</literal>.
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
- was used when building <productname>PostgreSQL</productname>.
+ was used when building <productname>PostgreSQL</productname>. If the
+ compression method has options they can be specified with the <literal>WITH
+ </literal> parameter.
The <literal>PRESERVE</literal> list contains a list of compression
methods used in the column and determines which of them may be kept.
Without <literal>PRESERVE</literal> or if any of the pre-existing
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index 5e13b3288c..78210c5461 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -69,7 +69,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI
GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] |
UNIQUE <replaceable class="parameter">index_parameters</replaceable> |
PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> |
- COMPRESSION <replaceable class="parameter">compression_method</replaceable> |
+ COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ WITH (<replaceable class="parameter">compression_options</replaceable>) ] |
REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ]
[ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] }
[ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -994,7 +994,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
</varlistentry>
<varlistentry>
- <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable></literal></term>
+ <term><literal>COMPRESSION <replaceable class="parameter">compression_method</replaceable> [ WITH (<replaceable class="parameter">compression_am_options</replaceable>) ]</literal></term>
<listitem>
<para>
The <literal>COMPRESSION</literal> clause sets the compression method
@@ -1011,6 +1011,8 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
Compression methods can be created with <xref
linkend="sql-create-access-method"/> or it can be set to any available
compression method.
+ If the compression method has options they can be specified with the
+ <literal>WITH </literal> parameter.
The default is <literal>pglz</literal>.
</para>
</listitem>
diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c
index 0ab5712c71..76f824bc6f 100644
--- a/src/backend/access/brin/brin_tuple.c
+++ b/src/backend/access/brin/brin_tuple.c
@@ -38,6 +38,7 @@
#include "access/toast_internals.h"
#include "access/tupdesc.h"
#include "access/tupmacs.h"
+#include "commands/defrem.h"
#include "utils/datum.h"
#include "utils/memutils.h"
@@ -215,8 +216,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple,
{
Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc,
keyno);
+ List *acoption = GetAttributeCompressionOptions(att);
Datum cvalue = toast_compress_datum(value,
- att->attcompression);
+ att->attcompression,
+ acoption);
if (DatumGetPointer(cvalue) != NULL)
{
diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c
index 1d43d5d2ff..0d3307e94f 100644
--- a/src/backend/access/common/indextuple.c
+++ b/src/backend/access/common/indextuple.c
@@ -22,6 +22,7 @@
#include "access/htup_details.h"
#include "access/itup.h"
#include "access/toast_internals.h"
+#include "commands/defrem.h"
/*
* This enables de-toasting of index entries. Needed until VACUUM is
@@ -104,8 +105,9 @@ index_form_tuple(TupleDesc tupleDescriptor,
(att->attstorage == TYPSTORAGE_EXTENDED ||
att->attstorage == TYPSTORAGE_MAIN))
{
+ List *acoption = GetAttributeCompressionOptions(att);
Datum cvalue = toast_compress_datum(untoasted_values[i],
- att->attcompression);
+ att->attcompression, acoption);
if (DatumGetPointer(cvalue) != NULL)
{
diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c
index c687d3ee9e..2dda6c038b 100644
--- a/src/backend/access/common/reloptions.c
+++ b/src/backend/access/common/reloptions.c
@@ -2112,3 +2112,67 @@ AlterTableGetRelOptionsLockLevel(List *defList)
return lockmode;
}
+
+/*
+ * Convert a DefElem list to the text array format that is used in
+ * pg_foreign_data_wrapper, pg_foreign_server, pg_user_mapping, and
+ * pg_foreign_table.
+ *
+ * Returns the array in the form of a Datum, or PointerGetDatum(NULL)
+ * if the list is empty.
+ *
+ * Note: The array is usually stored to database without further
+ * processing, hence any validation should be done before this
+ * conversion.
+ */
+Datum
+optionListToArray(List *options)
+{
+ ArrayBuildState *astate = NULL;
+ ListCell *cell;
+
+ foreach(cell, options)
+ {
+ DefElem *def = lfirst(cell);
+ const char *value;
+ Size len;
+ text *t;
+
+ value = defGetString(def);
+ len = VARHDRSZ + strlen(def->defname) + 1 + strlen(value);
+ t = palloc(len + 1);
+ SET_VARSIZE(t, len);
+ sprintf(VARDATA(t), "%s=%s", def->defname, value);
+
+ astate = accumArrayResult(astate, PointerGetDatum(t),
+ false, TEXTOID,
+ CurrentMemoryContext);
+ }
+
+ if (astate)
+ return makeArrayResult(astate, CurrentMemoryContext);
+
+ return PointerGetDatum(NULL);
+}
+
+/*
+ * Return human readable list of reloptions
+ */
+char *
+formatRelOptions(List *options)
+{
+ StringInfoData buf;
+ ListCell *cell;
+
+ initStringInfo(&buf);
+
+ foreach(cell, options)
+ {
+ DefElem *def = (DefElem *) lfirst(cell);
+
+ appendStringInfo(&buf, "%s%s=%s", buf.len > 0 ? ", " : "",
+ def->defname, defGetString(def));
+ }
+
+ return buf.data;
+}
diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c
index a3539065d3..5ed3312f39 100644
--- a/src/backend/access/common/toast_internals.c
+++ b/src/backend/access/common/toast_internals.c
@@ -44,10 +44,11 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid);
* ----------
*/
Datum
-toast_compress_datum(Datum value, Oid cmoid)
+toast_compress_datum(Datum value, Oid cmoid, List *cmoptions)
{
struct varlena *tmp = NULL;
int32 valsize;
+ void *options = NULL;
bool isCustomCompression = false;
const CompressionAmRoutine *cmroutine = NULL;
@@ -71,11 +72,20 @@ toast_compress_datum(Datum value, Oid cmoid)
break;
}
+ if (cmroutine->datum_initstate)
+ options = cmroutine->datum_initstate(cmoptions);
+
/* Call the actual compression function */
tmp = cmroutine->datum_compress((const struct varlena *)value,
isCustomCompression ?
TOAST_CUSTOM_COMPRESS_HDRSZ :
- TOAST_COMPRESS_HDRSZ);
+ TOAST_COMPRESS_HDRSZ, options);
+ if (options != NULL)
+ {
+ pfree(options);
+ list_free(cmoptions);
+ }
+
if (!tmp)
return PointerGetDatum(NULL);
diff --git a/src/backend/access/compression/compress_lz4.c b/src/backend/access/compression/compress_lz4.c
index 2a3c162836..7c422fc534 100644
--- a/src/backend/access/compression/compress_lz4.c
+++ b/src/backend/access/compression/compress_lz4.c
@@ -17,9 +17,73 @@
#endif
#include "access/compressamapi.h"
+#include "commands/defrem.h"
#include "fmgr.h"
#include "utils/builtins.h"
+/*
+ * Check options if specified. All validation is located here so
+ * we don't need to do it again in cminitstate function.
+ */
+static void
+lz4_cmcheck(List *options)
+{
+#ifndef HAVE_LIBLZ4
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#else
+ ListCell *lc;
+
+ foreach(lc, options)
+ {
+ DefElem *def = (DefElem *) lfirst(lc);
+
+ /*
+ * We don't need to check the range for the acceleration parameter
+ * because the LZ4_compress_fast will automatically replace the
+ * values <=0 with LZ4_ACCELERATION_DEFAULT (currently == 1). And,
+ * Values > LZ4_ACCELERATION_MAX with LZ4_ACCELERATION_MAX
+ * (currently == 65537c).
+ */
+ if (strcmp(def->defname, "acceleration") != 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_PARAMETER),
+ errmsg("unknown compression option for lz4: \"%s\"", def->defname)));
+ }
+#endif
+}
+
+static void *
+lz4_cminitstate(List *options)
+{
+#ifndef HAVE_LIBLZ4
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("not built with lz4 support")));
+#else
+ int32 *acceleration = palloc(sizeof(int32));
+
+ /* initialize with the default acceleration */
+ *acceleration = 1;
+
+ if (list_length(options) > 0)
+ {
+ ListCell *lc;
+
+ foreach(lc, options)
+ {
+ DefElem *def = (DefElem *) lfirst(lc);
+
+ if (strcmp(def->defname, "acceleration") == 0)
+ *acceleration = pg_atoi(defGetString(def), sizeof(int32), 0);
+ }
+ }
+
+ return acceleration;
+#endif
+}
+
/*
* lz4_cmcompress - compression routine for lz4 compression method
*
@@ -27,7 +91,7 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-lz4_cmcompress(const struct varlena *value, int32 header_size)
+lz4_cmcompress(const struct varlena *value, int32 header_size, void *options)
{
#ifndef HAVE_LIBLZ4
ereport(ERROR,
@@ -37,6 +101,7 @@ lz4_cmcompress(const struct varlena *value, int32 header_size)
int32 valsize;
int32 len;
int32 max_size;
+ int32 *acceleration = (int32 *) options;
struct varlena *tmp = NULL;
valsize = VARSIZE_ANY_EXHDR(value);
@@ -48,9 +113,9 @@ lz4_cmcompress(const struct varlena *value, int32 header_size)
max_size = LZ4_compressBound(valsize);
tmp = (struct varlena *) palloc(max_size + header_size);
- len = LZ4_compress_default(VARDATA_ANY(value),
- (char *) tmp + header_size,
- valsize, max_size);
+ len = LZ4_compress_fast(VARDATA_ANY(value),
+ (char *) tmp + header_size,
+ valsize, max_size, *acceleration);
if (len <= 0)
elog(ERROR, "could not compress data with lz4");
@@ -146,6 +211,8 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 header_size,
*/
const CompressionAmRoutine lz4_compress_methods = {
.type = T_CompressionAmRoutine,
+ .datum_check = lz4_cmcheck,
+ .datum_initstate = lz4_cminitstate,
.datum_compress = lz4_cmcompress,
.datum_decompress = lz4_cmdecompress,
.datum_decompress_slice = lz4_cmdecompress_slice
diff --git a/src/backend/access/compression/compress_pglz.c b/src/backend/access/compression/compress_pglz.c
index 7f6e7429fe..1ca912831b 100644
--- a/src/backend/access/compression/compress_pglz.c
+++ b/src/backend/access/compression/compress_pglz.c
@@ -14,11 +14,93 @@
#include "postgres.h"
#include "access/compressamapi.h"
+#include "access/toast_internals.h"
+#include "commands/defrem.h"
#include "common/pg_lzcompress.h"
#include "fmgr.h"
#include "utils/builtins.h"
+#define PGLZ_OPTIONS_COUNT 6
+
+static char *PGLZ_options[PGLZ_OPTIONS_COUNT] = {
+ "min_input_size",
+ "max_input_size",
+ "min_comp_rate",
+ "first_success_by",
+ "match_size_good",
+ "match_size_drop"
+};
+
+/*
+ * Convert value from reloptions to int32, and report if it is not correct.
+ * Also checks parameter names
+ */
+static int32
+parse_option(char *name, char *value)
+{
+ int i;
+
+ for (i = 0; i < PGLZ_OPTIONS_COUNT; i++)
+ {
+ if (strcmp(PGLZ_options[i], name) == 0)
+ return pg_atoi(value, 4, 0);
+ }
+
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_PARAMETER),
+ errmsg("unknown compression option for pglz: \"%s\"", name)));
+}
+
+/*
+ * Check PGLZ options if specified
+ */
+static void
+pglz_cmcheck(List *options)
+{
+ ListCell *lc;
+
+ foreach(lc, options)
+ {
+ DefElem *def = (DefElem *) lfirst(lc);
+
+ parse_option(def->defname, defGetString(def));
+ }
+}
+
+/*
+ * Configure PGLZ_Strategy struct for compression function
+ */
+static void *
+pglz_cminitstate(List *options)
+{
+ ListCell *lc;
+ PGLZ_Strategy *strategy = palloc(sizeof(PGLZ_Strategy));
+
+ /* initialize with default strategy values */
+ memcpy(strategy, PGLZ_strategy_default, sizeof(PGLZ_Strategy));
+ foreach(lc, options)
+ {
+ DefElem *def = (DefElem *) lfirst(lc);
+ int32 val = parse_option(def->defname, defGetString(def));
+
+ /* fill the strategy */
+ if (strcmp(def->defname, "min_input_size") == 0)
+ strategy->min_input_size = val;
+ else if (strcmp(def->defname, "max_input_size") == 0)
+ strategy->max_input_size = val;
+ else if (strcmp(def->defname, "min_comp_rate") == 0)
+ strategy->min_comp_rate = val;
+ else if (strcmp(def->defname, "first_success_by") == 0)
+ strategy->first_success_by = val;
+ else if (strcmp(def->defname, "match_size_good") == 0)
+ strategy->match_size_good = val;
+ else if (strcmp(def->defname, "match_size_drop") == 0)
+ strategy->match_size_drop = val;
+ }
+ return (void *) strategy;
+}
+
/*
* pglz_cmcompress - compression routine for pglz compression method
*
@@ -26,42 +108,41 @@
* compressed varlena, or NULL if compression fails.
*/
static struct varlena *
-pglz_cmcompress(const struct varlena *value, int32 header_size)
+pglz_cmcompress(const struct varlena *value, int32 header_size, void *options)
{
int32 valsize,
len;
- struct varlena *tmp = NULL;
+ struct varlena *tmp = NULL;
+ PGLZ_Strategy *strategy;
valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value));
+ strategy = (PGLZ_Strategy *) options;
/*
* No point in wasting a palloc cycle if value size is outside the allowed
* range for compression.
*/
- if (valsize < PGLZ_strategy_default->min_input_size ||
- valsize > PGLZ_strategy_default->max_input_size)
+ if (valsize < strategy->min_input_size ||
+ valsize > strategy->max_input_size)
return NULL;
- /*
- * Get maximum size of the compressed data that pglz compression may output
- * and allocate the memory for holding the compressed data and the header.
- */
tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
header_size);
- len = pglz_compress(VARDATA_ANY(value),
+ len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)),
valsize,
(char *) tmp + header_size,
- NULL);
- if (len < 0)
+ strategy);
+
+ if (len >= 0)
{
- pfree(tmp);
- return NULL;
+ SET_VARSIZE_COMPRESSED(tmp, len + header_size);
+ return tmp;
}
- SET_VARSIZE_COMPRESSED(tmp, len + header_size);
+ pfree(tmp);
- return tmp;
+ return NULL;
}
/*
@@ -125,10 +206,11 @@ pglz_cmdecompress_slice(const struct varlena *value, int32 header_size,
*/
const CompressionAmRoutine pglz_compress_methods = {
.type = T_CompressionAmRoutine,
+ .datum_check = pglz_cmcheck,
+ .datum_initstate = pglz_cminitstate,
.datum_compress = pglz_cmcompress,
.datum_decompress = pglz_cmdecompress,
- .datum_decompress_slice = pglz_cmdecompress_slice
-};
+ .datum_decompress_slice = pglz_cmdecompress_slice};
/* pglz compression handler function */
Datum
diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c
index 53f78f9c3e..c9d44c62fa 100644
--- a/src/backend/access/table/toast_helper.c
+++ b/src/backend/access/table/toast_helper.c
@@ -15,11 +15,13 @@
#include "postgres.h"
#include "access/detoast.h"
+#include "access/reloptions.h"
#include "access/table.h"
#include "access/toast_helper.h"
#include "access/toast_internals.h"
#include "catalog/pg_type_d.h"
-
+#include "commands/defrem.h"
+#include "utils/syscache.h"
/*
* Prepare to TOAST a tuple.
@@ -55,6 +57,7 @@ toast_tuple_init(ToastTupleContext *ttc)
ttc->ttc_attr[i].tai_colflags = 0;
ttc->ttc_attr[i].tai_oldexternal = NULL;
ttc->ttc_attr[i].tai_compression = att->attcompression;
+ ttc->ttc_attr[i].tai_cmoptions = GetAttributeCompressionOptions(att);
if (ttc->ttc_oldvalues != NULL)
{
@@ -230,7 +233,8 @@ toast_tuple_try_compression(ToastTupleContext *ttc, int attribute)
Datum new_value;
ToastAttrInfo *attr = &ttc->ttc_attr[attribute];
- new_value = toast_compress_datum(*value, attr->tai_compression);
+ new_value = toast_compress_datum(*value, attr->tai_compression,
+ attr->tai_cmoptions);
if (DatumGetPointer(new_value) != NULL)
{
diff --git a/src/backend/bootstrap/bootparse.y b/src/backend/bootstrap/bootparse.y
index 5fcd004e1b..a43e0e197c 100644
--- a/src/backend/bootstrap/bootparse.y
+++ b/src/backend/bootstrap/bootparse.y
@@ -239,6 +239,7 @@ Boot_CreateStmt:
true,
false,
InvalidOid,
+ NULL,
NULL);
elog(DEBUG4, "relation created with OID %u", id);
}
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index b53b6b50e6..18b04816f7 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -741,6 +741,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel,
TupleDesc tupdesc,
Oid new_rel_oid,
Datum *attoptions,
+ Datum *attcmoptions,
CatalogIndexState indstate)
{
TupleTableSlot **slot;
@@ -796,6 +797,11 @@ InsertPgAttributeTuples(Relation pg_attribute_rel,
else
slot[slotCount]->tts_isnull[Anum_pg_attribute_attoptions - 1] = true;
+ if (attcmoptions && attcmoptions[natts] != (Datum) 0)
+ slot[slotCount]->tts_values[Anum_pg_attribute_attcmoptions - 1] = attcmoptions[natts];
+ else
+ slot[slotCount]->tts_isnull[Anum_pg_attribute_attcmoptions - 1] = true;
+
/* start out with empty permissions and empty options */
slot[slotCount]->tts_isnull[Anum_pg_attribute_attacl - 1] = true;
slot[slotCount]->tts_isnull[Anum_pg_attribute_attfdwoptions - 1] = true;
@@ -843,6 +849,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel,
static void
AddNewAttributeTuples(Oid new_rel_oid,
TupleDesc tupdesc,
+ Datum *acoption,
char relkind)
{
Relation rel;
@@ -861,7 +868,8 @@ AddNewAttributeTuples(Oid new_rel_oid,
/* set stats detail level to a sane default */
for (int i = 0; i < natts; i++)
tupdesc->attrs[i].attstattarget = -1;
- InsertPgAttributeTuples(rel, tupdesc, new_rel_oid, NULL, indstate);
+ InsertPgAttributeTuples(rel, tupdesc, new_rel_oid,
+ NULL, acoption, indstate);
/* add dependencies on their datatypes and collations */
for (int i = 0; i < natts; i++)
@@ -893,7 +901,7 @@ AddNewAttributeTuples(Oid new_rel_oid,
td = CreateTupleDesc(lengthof(SysAtt), (FormData_pg_attribute **) &SysAtt);
- InsertPgAttributeTuples(rel, td, new_rel_oid, NULL, indstate);
+ InsertPgAttributeTuples(rel, td, new_rel_oid, NULL, NULL, indstate);
FreeTupleDesc(td);
}
@@ -1160,6 +1168,7 @@ heap_create_with_catalog(const char *relname,
bool allow_system_table_mods,
bool is_internal,
Oid relrewrite,
+ Datum *acoptions,
ObjectAddress *typaddress)
{
Relation pg_class_desc;
@@ -1418,7 +1427,7 @@ heap_create_with_catalog(const char *relname,
/*
* now add tuples to pg_attribute for the attributes in our new relation.
*/
- AddNewAttributeTuples(relid, new_rel_desc->rd_att, relkind);
+ AddNewAttributeTuples(relid, new_rel_desc->rd_att, acoptions, relkind);
/*
* Make a dependency link to force the relation to be deleted if its
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 0db2a7430f..ce59075ffd 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -108,10 +108,12 @@ static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
List *indexColNames,
Oid accessMethodObjectId,
Oid *collationObjectId,
- Oid *classObjectId);
+ Oid *classObjectId,
+ Datum *acoptions);
static void InitializeAttributeOids(Relation indexRelation,
int numatts, Oid indexoid);
-static void AppendAttributeTuples(Relation indexRelation, Datum *attopts);
+static void AppendAttributeTuples(Relation indexRelation, Datum *attopts,
+ Datum *attcmopts);
static void UpdateIndexRelation(Oid indexoid, Oid heapoid,
Oid parentIndexId,
IndexInfo *indexInfo,
@@ -273,7 +275,8 @@ ConstructTupleDescriptor(Relation heapRelation,
List *indexColNames,
Oid accessMethodObjectId,
Oid *collationObjectId,
- Oid *classObjectId)
+ Oid *classObjectId,
+ Datum *acoptions)
{
int numatts = indexInfo->ii_NumIndexAttrs;
int numkeyatts = indexInfo->ii_NumIndexKeyAttrs;
@@ -334,6 +337,9 @@ ConstructTupleDescriptor(Relation heapRelation,
{
/* Simple index column */
const FormData_pg_attribute *from;
+ HeapTuple attr_tuple;
+ Datum attcmoptions;
+ bool isNull;
Assert(atnum > 0); /* should've been caught above */
@@ -350,6 +356,23 @@ ConstructTupleDescriptor(Relation heapRelation,
to->attstorage = from->attstorage;
to->attalign = from->attalign;
to->attcompression = from->attcompression;
+
+ attr_tuple = SearchSysCache2(ATTNUM,
+ ObjectIdGetDatum(from->attrelid),
+ Int16GetDatum(from->attnum));
+ if (!HeapTupleIsValid(attr_tuple))
+ elog(ERROR, "cache lookup failed for attribute %d of relation %u",
+ from->attnum, from->attrelid);
+
+ attcmoptions = SysCacheGetAttr(ATTNUM, attr_tuple,
+ Anum_pg_attribute_attcmoptions,
+ &isNull);
+ if (isNull)
+ acoptions[i] = PointerGetDatum(NULL);
+ else
+ acoptions[i] =
+ PointerGetDatum(DatumGetArrayTypePCopy(attcmoptions));
+ ReleaseSysCache(attr_tuple);
}
else
{
@@ -490,7 +513,7 @@ InitializeAttributeOids(Relation indexRelation,
* ----------------------------------------------------------------
*/
static void
-AppendAttributeTuples(Relation indexRelation, Datum *attopts)
+AppendAttributeTuples(Relation indexRelation, Datum *attopts, Datum *attcmopts)
{
Relation pg_attribute;
CatalogIndexState indstate;
@@ -508,7 +531,8 @@ AppendAttributeTuples(Relation indexRelation, Datum *attopts)
*/
indexTupDesc = RelationGetDescr(indexRelation);
- InsertPgAttributeTuples(pg_attribute, indexTupDesc, InvalidOid, attopts, indstate);
+ InsertPgAttributeTuples(pg_attribute, indexTupDesc, InvalidOid,
+ attopts, attcmopts, indstate);
CatalogCloseIndexes(indstate);
@@ -721,6 +745,7 @@ index_create(Relation heapRelation,
bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0;
bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0;
char relkind;
+ Datum *acoptions;
TransactionId relfrozenxid;
MultiXactId relminmxid;
@@ -876,6 +901,8 @@ index_create(Relation heapRelation,
indexRelationName, RelationGetRelationName(heapRelation))));
}
+ acoptions = (Datum *) palloc0(sizeof(Datum) * indexInfo->ii_NumIndexAttrs);
+
/*
* construct tuple descriptor for index tuples
*/
@@ -884,7 +911,8 @@ index_create(Relation heapRelation,
indexColNames,
accessMethodObjectId,
collationObjectId,
- classObjectId);
+ classObjectId,
+ acoptions);
/*
* Allocate an OID for the index, unless we were told what to use.
@@ -975,7 +1003,8 @@ index_create(Relation heapRelation,
/*
* append ATTRIBUTE tuples for the index
*/
- AppendAttributeTuples(indexRelation, indexInfo->ii_OpclassOptions);
+ AppendAttributeTuples(indexRelation, indexInfo->ii_OpclassOptions,
+ acoptions);
/* ----------------
* update pg_index
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c
index a549481557..348b7d3a37 100644
--- a/src/backend/catalog/toasting.c
+++ b/src/backend/catalog/toasting.c
@@ -260,6 +260,7 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
true,
true,
InvalidOid,
+ NULL,
NULL);
Assert(toast_relid != InvalidOid);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 096a06f7b3..0ad946d802 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -699,6 +699,7 @@ make_new_heap(Oid OIDOldHeap, Oid NewTableSpace, char relpersistence,
true,
true,
OIDOldHeap,
+ NULL,
NULL);
Assert(OIDNewHeap != InvalidOid);
diff --git a/src/backend/commands/compressioncmds.c b/src/backend/commands/compressioncmds.c
index fd6db24e7f..160d64ad32 100644
--- a/src/backend/commands/compressioncmds.c
+++ b/src/backend/commands/compressioncmds.c
@@ -29,6 +29,7 @@
#include "utils/builtins.h"
#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
+#include "utils/syscache.h"
/*
* Get list of all supported compression methods for the given attribute.
@@ -181,7 +182,7 @@ BinaryUpgradeAddPreserve(Form_pg_attribute att, List *preserve)
*/
Oid
GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
- bool *need_rewrite)
+ Datum *acoptions, bool *need_rewrite)
{
Oid cmoid;
char typstorage = get_typstorage(att->atttypid);
@@ -215,6 +216,20 @@ GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
errmsg("not built with lz4 support")));
#endif
+ /* if compression options are given then check them */
+ if (compression->options)
+ {
+ CompressionAmRoutine *routine = GetCompressionAmRoutineByAmId(cmoid);
+
+ /* we need routine only to call cmcheck function */
+ if (routine->datum_check != NULL)
+ routine->datum_check(compression->options);
+
+ *acoptions = optionListToArray(compression->options);
+ }
+ else
+ *acoptions = PointerGetDatum(NULL);
+
/*
* Determine if the column needs rewrite or not. Rewrite conditions: SET
* COMPRESSION without PRESERVE - SET COMPRESSION with PRESERVE but not
@@ -286,15 +301,71 @@ GetAttributeCompression(Form_pg_attribute att, ColumnCompression *compression,
* Construct ColumnCompression node from the compression method oid.
*/
ColumnCompression *
-MakeColumnCompression(Oid attcompression)
+MakeColumnCompression(Form_pg_attribute att)
{
ColumnCompression *node;
- if (!OidIsValid(attcompression))
+ if (!OidIsValid(att->attcompression))
return NULL;
node = makeNode(ColumnCompression);
- node->cmname = get_am_name(attcompression);
+ node->cmname = get_am_name(att->attcompression);
+ node->options = GetAttributeCompressionOptions(att);
return node;
}
+
+/*
+ * Fetch atttribute's compression options
+ */
+List *
+GetAttributeCompressionOptions(Form_pg_attribute att)
+{
+ HeapTuple attr_tuple;
+ Datum attcmoptions;
+ List *acoptions;
+ bool isNull;
+
+ attr_tuple = SearchSysCache2(ATTNUM,
+ ObjectIdGetDatum(att->attrelid),
+ Int16GetDatum(att->attnum));
+ if (!HeapTupleIsValid(attr_tuple))
+ elog(ERROR, "cache lookup failed for attribute %d of relation %u",
+ att->attnum, att->attrelid);
+
+ attcmoptions = SysCacheGetAttr(ATTNUM, attr_tuple,
+ Anum_pg_attribute_attcmoptions,
+ &isNull);
+ if (isNull)
+ acoptions = NULL;
+ else
+ acoptions = untransformRelOptions(attcmoptions);
+
+ ReleaseSysCache(attr_tuple);
+
+ return acoptions;
+}
+
+/*
+ * Compare compression options for two columns.
+ */
+void
+CheckCompressionMismatch(ColumnCompression *c1, ColumnCompression *c2,
+ const char *attributeName)
+{
+ if (strcmp(c1->cmname, c2->cmname))
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("column \"%s\" has a compression method conflict",
+ attributeName),
+ errdetail("%s versus %s", c1->cmname, c2->cmname)));
+
+ if (!equal(c1->options, c2->options))
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("column \"%s\" has a compression options conflict",
+ attributeName),
+ errdetail("(%s) versus (%s)",
+ formatRelOptions(c1->options),
+ formatRelOptions(c2->options))));
+}
diff --git a/src/backend/commands/foreigncmds.c b/src/backend/commands/foreigncmds.c
index eb7103fd3b..ae9ae2c51b 100644
--- a/src/backend/commands/foreigncmds.c
+++ b/src/backend/commands/foreigncmds.c
@@ -49,50 +49,6 @@ typedef struct
/* Internal functions */
static void import_error_callback(void *arg);
-
-/*
- * Convert a DefElem list to the text array format that is used in
- * pg_foreign_data_wrapper, pg_foreign_server, pg_user_mapping, and
- * pg_foreign_table.
- *
- * Returns the array in the form of a Datum, or PointerGetDatum(NULL)
- * if the list is empty.
- *
- * Note: The array is usually stored to database without further
- * processing, hence any validation should be done before this
- * conversion.
- */
-static Datum
-optionListToArray(List *options)
-{
- ArrayBuildState *astate = NULL;
- ListCell *cell;
-
- foreach(cell, options)
- {
- DefElem *def = lfirst(cell);
- const char *value;
- Size len;
- text *t;
-
- value = defGetString(def);
- len = VARHDRSZ + strlen(def->defname) + 1 + strlen(value);
- t = palloc(len + 1);
- SET_VARSIZE(t, len);
- sprintf(VARDATA(t), "%s=%s", def->defname, value);
-
- astate = accumArrayResult(astate, PointerGetDatum(t),
- false, TEXTOID,
- CurrentMemoryContext);
- }
-
- if (astate)
- return makeArrayResult(astate, CurrentMemoryContext);
-
- return PointerGetDatum(NULL);
-}
-
-
/*
* Transform a list of DefElem into text array format. This is substantially
* the same thing as optionListToArray(), except we recognize SET/ADD/DROP
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 5fba7352a3..b72da302f9 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -609,6 +609,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
LOCKMODE parentLockmode;
const char *accessMethod = NULL;
Oid accessMethodId = InvalidOid;
+ Datum *acoptions;
/*
* Truncate relname to appropriate length (probably a waste of time, as
@@ -813,6 +814,8 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
cookedDefaults = NIL;
attnum = 0;
+ acoptions = (Datum *) palloc0(sizeof(Datum) * descriptor->natts);
+
foreach(listptr, stmt->tableElts)
{
ColumnDef *colDef = lfirst(listptr);
@@ -866,8 +869,9 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
if (relkind == RELKIND_RELATION ||
relkind == RELKIND_PARTITIONED_TABLE ||
relkind == RELKIND_MATVIEW)
- attr->attcompression =
- GetAttributeCompression(attr, colDef->compression, NULL);
+ attr->attcompression = GetAttributeCompression(attr,
+ colDef->compression,
+ &acoptions[attnum - 1], NULL);
else
attr->attcompression = InvalidOid;
}
@@ -921,8 +925,11 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
allowSystemTableMods,
false,
InvalidOid,
+ acoptions,
typaddress);
+ pfree(acoptions);
+
/*
* We must bump the command counter to make the newly-created relation
* tuple visible for opening.
@@ -2432,16 +2439,14 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
if (OidIsValid(attribute->attcompression))
{
ColumnCompression *compression =
- MakeColumnCompression(attribute->attcompression);
+ MakeColumnCompression(attribute);
if (!def->compression)
def->compression = compression;
- else if (strcmp(def->compression->cmname, compression->cmname))
- ereport(ERROR,
- (errcode(ERRCODE_DATATYPE_MISMATCH),
- errmsg("column \"%s\" has a compression method conflict",
- attributeName),
- errdetail("%s versus %s", def->compression->cmname, compression->cmname)));
+ else
+ CheckCompressionMismatch(def->compression,
+ compression,
+ attributeName);
}
def->inhcount++;
@@ -2478,8 +2483,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collOid = attribute->attcollation;
def->constraints = NIL;
def->location = -1;
- def->compression = MakeColumnCompression(
- attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute);
inhSchema = lappend(inhSchema, def);
newattmap->attnums[parent_attno - 1] = ++child_attno;
}
@@ -2729,14 +2733,9 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
if (!def->compression)
def->compression = newdef->compression;
else if (newdef->compression)
- {
- if (strcmp(def->compression->cmname, newdef->compression->cmname))
- ereport(ERROR,
- (errcode(ERRCODE_DATATYPE_MISMATCH),
- errmsg("column \"%s\" has a compression method conflict",
- attributeName),
- errdetail("%s versus %s", def->compression->cmname, newdef->compression->cmname)));
- }
+ CheckCompressionMismatch(def->compression,
+ newdef->compression,
+ attributeName);
/* Mark the column as locally defined */
def->is_local = true;
@@ -6271,6 +6270,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
AclResult aclresult;
ObjectAddress address;
TupleDesc tupdesc;
+ Datum acoptions = PointerGetDatum(NULL);
FormData_pg_attribute *aattr[] = {&attribute};
/* At top level, permission check was done in ATPrepCmd, else do it */
@@ -6434,6 +6434,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
attribute.attcompression = GetAttributeCompression(&attribute,
colDef->compression,
+ &acoptions,
NULL);
else
attribute.attcompression = InvalidOid;
@@ -6444,7 +6445,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
tupdesc = CreateTupleDesc(lengthof(aattr), (FormData_pg_attribute **) &aattr);
- InsertPgAttributeTuples(attrdesc, tupdesc, myrelid, NULL, NULL);
+ InsertPgAttributeTuples(attrdesc, tupdesc, myrelid, NULL, &acoptions, NULL);
table_close(attrdesc, RowExclusiveLock);
@@ -7840,13 +7841,20 @@ ATExecSetOptions(Relation rel, const char *colName, Node *options,
* the respective input values are valid.
*/
static void
-ApplyChangesToIndexes(Relation rel, Relation attrelation, AttrNumber attnum,
- Oid newcompression, char newstorage, LOCKMODE lockmode)
+ApplyChangesToIndexes(Relation rel, Relation attrel, AttrNumber attnum,
+ Oid newcompression, char newstorage, Datum acoptions,
+ LOCKMODE lockmode)
{
HeapTuple tuple;
ListCell *lc;
Form_pg_attribute attrtuple;
+ /*
+ * Compression option can only be valid if we are updating the compression
+ * method.
+ */
+ Assert(DatumGetPointer(acoptions) == NULL || OidIsValid(newcompression));
+
foreach(lc, RelationGetIndexList(rel))
{
Oid indexoid = lfirst_oid(lc);
@@ -7882,7 +7890,29 @@ ApplyChangesToIndexes(Relation rel, Relation attrelation, AttrNumber attnum,
if (newstorage != '\0')
attrtuple->attstorage = newstorage;
- CatalogTupleUpdate(attrelation, &tuple->t_self, tuple);
+ if (DatumGetPointer(acoptions) != NULL)
+ {
+ Datum values[Natts_pg_attribute];
+ bool nulls[Natts_pg_attribute];
+ bool replace[Natts_pg_attribute];
+ HeapTuple newtuple;
+
+ /* Initialize buffers for new tuple values */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replace, false, sizeof(replace));
+
+ values[Anum_pg_attribute_attcmoptions - 1] = acoptions;
+ replace[Anum_pg_attribute_attcmoptions - 1] = true;
+
+ newtuple = heap_modify_tuple(tuple, RelationGetDescr(attrel),
+ values, nulls, replace);
+ CatalogTupleUpdate(attrel, &newtuple->t_self, newtuple);
+
+ heap_freetuple(newtuple);
+ }
+ else
+ CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
InvokeObjectPostAlterHook(RelationRelationId,
RelationGetRelid(rel),
@@ -7973,7 +8003,7 @@ ATExecSetStorage(Relation rel, const char *colName, Node *newValue, LOCKMODE loc
* matching behavior of index.c ConstructTupleDescriptor()).
*/
ApplyChangesToIndexes(rel, attrelation, attnum, InvalidOid, newstorage,
- lockmode);
+ PointerGetDatum(NULL), lockmode);
table_close(attrelation, RowExclusiveLock);
@@ -15145,9 +15175,11 @@ ATExecSetCompression(AlteredTableInfo *tab,
char typstorage;
Oid cmoid;
bool need_rewrite;
+ HeapTuple newtuple;
Datum values[Natts_pg_attribute];
bool nulls[Natts_pg_attribute];
bool replace[Natts_pg_attribute];
+ Datum acoptions;
ObjectAddress address;
attrel = table_open(AttributeRelationId, RowExclusiveLock);
@@ -15182,7 +15214,8 @@ ATExecSetCompression(AlteredTableInfo *tab,
memset(replace, false, sizeof(replace));
/* Get the attribute compression method. */
- cmoid = GetAttributeCompression(atttableform, compression, &need_rewrite);
+ cmoid = GetAttributeCompression(atttableform, compression, &acoptions,
+ &need_rewrite);
if (atttableform->attcompression != cmoid)
add_column_compression_dependency(atttableform->attrelid,
@@ -15192,8 +15225,17 @@ ATExecSetCompression(AlteredTableInfo *tab,
atttableform->attcompression = cmoid;
- atttableform->attcompression = cmoid;
- CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
+ /* update an existing entry */
+ if (acoptions)
+ {
+ values[Anum_pg_attribute_attcmoptions - 1] = acoptions;
+ replace[Anum_pg_attribute_attcmoptions - 1] = true;
+ newtuple = heap_modify_tuple(tuple, RelationGetDescr(attrel),
+ values, nulls, replace);
+ CatalogTupleUpdate(attrel, &newtuple->t_self, newtuple);
+ }
+ else
+ CatalogTupleUpdate(attrel, &tuple->t_self, tuple);
InvokeObjectPostAlterHook(RelationRelationId,
RelationGetRelid(rel),
@@ -15201,8 +15243,10 @@ ATExecSetCompression(AlteredTableInfo *tab,
ReleaseSysCache(tuple);
- /* apply changes to the index column as well */
- ApplyChangesToIndexes(rel, attrel, attnum, cmoid, '\0', lockmode);
+ /* Apply the change to indexes as well */
+ ApplyChangesToIndexes(rel, attrel, attnum, cmoid, '\0', acoptions,
+ lockmode);
+
table_close(attrel, RowExclusiveLock);
/* make changes visible */
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 6a11f8eb60..991340e2ad 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2993,6 +2993,7 @@ _copyColumnCompression(const ColumnCompression *from)
COPY_STRING_FIELD(cmname);
COPY_SCALAR_FIELD(preserve_all);
+ COPY_NODE_FIELD(options);
COPY_NODE_FIELD(preserve);
return newnode;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 26a9b85974..57a2975da1 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2624,6 +2624,7 @@ _equalColumnCompression(const ColumnCompression *a, const ColumnCompression *b)
{
COMPARE_STRING_FIELD(cmname);
COMPARE_SCALAR_FIELD(preserve_all);
+ COMPARE_NODE_FIELD(options);
COMPARE_NODE_FIELD(preserve);
return true;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index b584a58ba3..bc5c64df1e 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2888,6 +2888,7 @@ _outColumnCompression(StringInfo str, const ColumnCompression *node)
WRITE_STRING_FIELD(cmname);
WRITE_BOOL_FIELD(preserve_all);
+ WRITE_NODE_FIELD(options);
WRITE_NODE_FIELD(preserve);
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index b22ce818cd..00fe3b7a5f 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -415,6 +415,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
relation_expr_list dostmt_opt_list
transform_element_list transform_type_list
TriggerTransitions TriggerReferencing
+ optCompressionParameters
vacuum_relation_list opt_vacuum_relation_list
drop_option_list
@@ -3503,11 +3504,17 @@ compressionClause:
COMPRESSION name { $$ = pstrdup($2); }
;
+optCompressionParameters:
+ WITH '(' generic_option_list ')' { $$ = $3; }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
optColumnCompression:
- compressionClause
+ compressionClause optCompressionParameters
{
ColumnCompression *n = makeNode(ColumnCompression);
n->cmname = $1;
+ n->options = (List *) $2;
n->preserve = NIL;
$$ = (Node *) n;
}
@@ -3515,14 +3522,15 @@ optColumnCompression:
;
alterColumnCompression:
- compressionClause optCompressionPreserve
+ compressionClause optCompressionParameters optCompressionPreserve
{
ColumnCompression *n = makeNode(ColumnCompression);
n->cmname = $1;
- n->preserve = (List *) $2;
+ n->options = (List *) $2;
+ n->preserve = (List *) $3;
$$ = (Node *) n;
}
- | compressionClause PRESERVE ALL
+ | compressionClause optCompressionParameters PRESERVE ALL
{
ColumnCompression *n = makeNode(ColumnCompression);
n->cmname = $1;
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index f50b50cc8e..24e192175e 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1086,7 +1086,7 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla
/* Likewise, copy compression if requested */
if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0
&& OidIsValid(attribute->attcompression))
- def->compression = MakeColumnCompression(attribute->attcompression);
+ def->compression = MakeColumnCompression(attribute);
else
def->compression = NULL;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 88fa7e1ed3..f9517549f9 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -8709,10 +8709,17 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
if (createWithCompression)
appendPQExpBuffer(q,
- "am.amname AS attcmname,\n");
+ "am.amname AS attcmname,\n"
+ "pg_catalog.array_to_string(ARRAY("
+ "SELECT pg_catalog.quote_ident(option_name) || "
+ "' ' || pg_catalog.quote_literal(option_value) "
+ "FROM pg_catalog.pg_options_to_table(a.attcmoptions) "
+ "ORDER BY option_name"
+ "), E',\n ') AS attcmoptions,\n");
else
appendPQExpBuffer(q,
- "NULL AS attcmname,\n");
+ "NULL AS attcmname,\n"
+ "NULL AS attcmoptions,\n");
if (fout->remoteVersion >= 110000)
appendPQExpBufferStr(q,
@@ -8765,6 +8772,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *));
tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *));
tbinfo->attcmnames = (char **) pg_malloc(ntups * sizeof(char *));
+ tbinfo->attcmoptions = (char **) pg_malloc(ntups * sizeof(char *));
tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool));
tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *));
@@ -8794,6 +8802,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables)
tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions")));
tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval")));
tbinfo->attcmnames[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmname")));
+ tbinfo->attcmoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attcmoptions")));
tbinfo->attrdefs[j] = NULL; /* fix below */
if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't')
hasdefaults = true;
@@ -15959,7 +15968,8 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
continue;
has_non_default_compression = (tbinfo->attcmnames[j] &&
- (strcmp(tbinfo->attcmnames[j], "pglz") != 0));
+ ((strcmp(tbinfo->attcmnames[j], "pglz") != 0) ||
+ nonemptyReloptions(tbinfo->attcmoptions[j])));
/* Format properly if not first attr */
if (actual_atts == 0)
@@ -16005,6 +16015,10 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
{
appendPQExpBuffer(q, " COMPRESSION %s",
tbinfo->attcmnames[j]);
+
+ if (nonemptyReloptions(tbinfo->attcmoptions[j]))
+ appendPQExpBuffer(q, " WITH (%s)",
+ tbinfo->attcmoptions[j]);
}
if (print_default)
@@ -16436,6 +16450,9 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo)
appendPQExpBuffer(q, "ALTER TABLE %s ALTER COLUMN %s\nSET COMPRESSION %s",
qualrelname, fmtId(tbinfo->attnames[j]), tbinfo->attcmnames[j]);
+ if (nonemptyReloptions(tbinfo->attcmoptions[j]))
+ appendPQExpBuffer(q, "\nWITH (%s) ", tbinfo->attcmoptions[j]);
+
if (cminfo->nitems > 0)
{
appendPQExpBuffer(q, "\nPRESERVE (");
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index a829528cd0..da47f3173f 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -327,6 +327,7 @@ typedef struct _tableInfo
bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */
char *amname; /* relation access method */
char **attcmnames; /* per-attribute current compression method */
+ char **attcmoptions; /* per-attribute compression options */
struct _attrCompressionInfo **attcompression; /* per-attribute all
compression data */
/*
diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h
index bac7b47cfc..7f3346b2a6 100644
--- a/src/include/access/compressamapi.h
+++ b/src/include/access/compressamapi.h
@@ -51,8 +51,11 @@ extern Oid GetDefaultToastCompression(void);
#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \
(storage) != TYPSTORAGE_EXTERNAL)
/* compression handler routines */
+typedef void (*cmcheck_function) (List *options);
+typedef void *(*cminitstate_function) (List *options);
typedef struct varlena *(*cmcompress_function) (const struct varlena *value,
- int32 toast_header_size);
+ int32 toast_header_size,
+ void *options);
typedef struct varlena *(*cmdecompress_function) (const struct varlena *value,
int32 toast_header_size);
typedef struct varlena *(*cmdecompress_slice_function)
@@ -63,14 +66,25 @@ typedef struct varlena *(*cmdecompress_slice_function)
/*
* API struct for a compression AM.
*
+ * 'cmcheck' - called when attribute is linking with compression method.
+ * This function should check compability of compression method with
+ * the attribute and its options.
+ *
+ * 'cminitstate' - called when CompressionAmOptions instance is created.
+ * Should return pointer to a memory in a caller memory context, or NULL.
+ * Could be used to pass some internal state between compression function
+ * calls, like internal structure for parsed compression options.
+ *
* 'datum_compress' - varlena compression function.
* 'datum_decompress' - varlena decompression function.
* 'datum_decompress_slice' - varlena slice decompression functions.
*/
typedef struct CompressionAmRoutine
{
- NodeTag type;
+ NodeTag type;
+ cmcheck_function datum_check; /* can be NULL */
+ cminitstate_function datum_initstate; /* can be NULL */
cmcompress_function datum_compress;
cmdecompress_function datum_decompress;
cmdecompress_slice_function datum_decompress_slice;
diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h
index dca0bc37f3..fe8de405ac 100644
--- a/src/include/access/toast_helper.h
+++ b/src/include/access/toast_helper.h
@@ -15,6 +15,7 @@
#define TOAST_HELPER_H
#include "utils/rel.h"
+#include "nodes/pg_list.h"
/*
* Information about one column of a tuple being toasted.
@@ -33,6 +34,7 @@ typedef struct
int32 tai_size;
uint8 tai_colflags;
Oid tai_compression;
+ List *tai_cmoptions;
} ToastAttrInfo;
/*
diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h
index ac28f9ed55..864de31739 100644
--- a/src/include/access/toast_internals.h
+++ b/src/include/access/toast_internals.h
@@ -54,7 +54,7 @@ typedef struct toast_compress_header_custom
#define TOAST_COMPRESS_SET_CMOID(ptr, oid) \
(((toast_compress_header_custom *)(ptr))->cmoid = (oid))
-extern Datum toast_compress_datum(Datum value, Oid cmoid);
+extern Datum toast_compress_datum(Datum value, Oid cmoid, List *cmoptions);
extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock);
extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative);
diff --git a/src/include/catalog/heap.h b/src/include/catalog/heap.h
index 6ce480b49c..5ecb6f3653 100644
--- a/src/include/catalog/heap.h
+++ b/src/include/catalog/heap.h
@@ -81,6 +81,7 @@ extern Oid heap_create_with_catalog(const char *relname,
bool allow_system_table_mods,
bool is_internal,
Oid relrewrite,
+ Datum *acoptions,
ObjectAddress *typaddress);
extern void heap_drop_with_catalog(Oid relid);
@@ -97,6 +98,7 @@ extern void InsertPgAttributeTuples(Relation pg_attribute_rel,
TupleDesc tupdesc,
Oid new_rel_oid,
Datum *attoptions,
+ Datum *attcmoptions,
CatalogIndexState indstate);
extern void InsertPgClassTuple(Relation pg_class_desc,
diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h
index 797e78b17c..7c74bc314e 100644
--- a/src/include/catalog/pg_attribute.h
+++ b/src/include/catalog/pg_attribute.h
@@ -178,6 +178,9 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75,
/* Column-level FDW options */
text attfdwoptions[1] BKI_DEFAULT(_null_);
+ /* current compression options */
+ text attcmoptions[1] BKI_DEFAULT(_null_);
+
/*
* Missing value for added columns. This is a one element array which lets
* us store a value of the attribute type here.
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index bd53f9bb0f..8271e97dea 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -134,6 +134,8 @@ extern Datum transformGenericOptions(Oid catalogId,
Datum oldOptions,
List *options,
Oid fdwvalidator);
+extern Datum optionListToArray(List *options);
+extern char *formatRelOptions(List *options);
/* commands/amcmds.c */
extern ObjectAddress CreateAccessMethod(CreateAmStmt *stmt);
@@ -146,9 +148,12 @@ extern char *get_am_name(Oid amOid);
/* commands/compressioncmds.c */
extern Oid GetAttributeCompression(Form_pg_attribute att,
ColumnCompression *compression,
- bool *need_rewrite);
-extern ColumnCompression *MakeColumnCompression(Oid atttcompression);
+ Datum *acoptions, bool *need_rewrite);
+extern ColumnCompression *MakeColumnCompression(Form_pg_attribute att);
extern bool IsCompressionSupported(Form_pg_attribute att, Oid cmoid);
+extern List *GetAttributeCompressionOptions(Form_pg_attribute att);
+extern void CheckCompressionMismatch(ColumnCompression *c1, ColumnCompression *c2,
+ const char *attributeName);
/* support routines in commands/define.c */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index ce0913e18a..c3c878b80e 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -634,6 +634,7 @@ typedef struct ColumnCompression
NodeTag type;
char *cmname;
bool preserve_all;
+ List *options;
List *preserve;
} ColumnCompression;
diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out
index e52e272d39..4af72bdbdc 100644
--- a/src/test/regress/expected/compression.out
+++ b/src/test/regress/expected/compression.out
@@ -297,6 +297,58 @@ SELECT pg_column_compression(f1) FROM cmdata;
Indexes:
"idx" btree (f1)
+-- compression options
+CREATE TABLE cmdata3(f1 TEXT COMPRESSION pglz WITH (min_input_size '100'));
+CREATE INDEX idx1 ON cmdata3(f1);
+INSERT INTO cmdata3 VALUES(repeat('1234567890',1000));
+SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'cmdata3'::regclass AND attname='f1';
+ attcmoptions
+----------------------
+ {min_input_size=100}
+(1 row)
+
+SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'idx1'::regclass AND attname='f1';
+ attcmoptions
+----------------------
+ {min_input_size=100}
+(1 row)
+
+ALTER TABLE cmdata3 ALTER COLUMN f1 SET COMPRESSION lz4 WITH (acceleration '50');
+INSERT INTO cmdata3 VALUES(repeat('1234567890',1004));
+SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'cmdata3'::regclass AND attname='f1';
+ attcmoptions
+-------------------
+ {acceleration=50}
+(1 row)
+
+SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'idx1'::regclass AND attname='f1';
+ attcmoptions
+-------------------
+ {acceleration=50}
+(1 row)
+
+ALTER TABLE cmdata3 ALTER COLUMN f1 SET COMPRESSION pglz WITH (min_input_size '200') PRESERVE (lz4);
+INSERT INTO cmdata3 VALUES(repeat('1234567890',1008));
+SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'cmdata3'::regclass AND attname='f1';
+ attcmoptions
+----------------------
+ {min_input_size=200}
+(1 row)
+
+SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'idx1'::regclass AND attname='f1';
+ attcmoptions
+----------------------
+ {min_input_size=200}
+(1 row)
+
+SELECT pg_column_compression(f1) FROM cmdata3;
+ pg_column_compression
+-----------------------
+ lz4
+ lz4
+ pglz
+(3 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out
index d4e215c931..211fbfc0bb 100644
--- a/src/test/regress/expected/compression_1.out
+++ b/src/test/regress/expected/compression_1.out
@@ -279,6 +279,60 @@ SELECT pg_column_compression(f1) FROM cmdata;
Indexes:
"idx" btree (f1)
+-- compression options
+CREATE TABLE cmdata3(f1 TEXT COMPRESSION pglz WITH (min_input_size '100'));
+CREATE INDEX idx1 ON cmdata3(f1);
+INSERT INTO cmdata3 VALUES(repeat('1234567890',1000));
+SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'cmdata3'::regclass AND attname='f1';
+ attcmoptions
+----------------------
+ {min_input_size=100}
+(1 row)
+
+SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'idx1'::regclass AND attname='f1';
+ attcmoptions
+----------------------
+ {min_input_size=100}
+(1 row)
+
+ALTER TABLE cmdata3 ALTER COLUMN f1 SET COMPRESSION lz4 WITH (acceleration '50');
+ERROR: not built with lz4 support
+INSERT INTO cmdata3 VALUES(repeat('1234567890',1004));
+SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'cmdata3'::regclass AND attname='f1';
+ attcmoptions
+----------------------
+ {min_input_size=100}
+(1 row)
+
+SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'idx1'::regclass AND attname='f1';
+ attcmoptions
+----------------------
+ {min_input_size=100}
+(1 row)
+
+ALTER TABLE cmdata3 ALTER COLUMN f1 SET COMPRESSION pglz WITH (min_input_size '200') PRESERVE (lz4);
+ERROR: "lz4" compression access method cannot be preserved
+INSERT INTO cmdata3 VALUES(repeat('1234567890',1008));
+SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'cmdata3'::regclass AND attname='f1';
+ attcmoptions
+----------------------
+ {min_input_size=100}
+(1 row)
+
+SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'idx1'::regclass AND attname='f1';
+ attcmoptions
+----------------------
+ {min_input_size=100}
+(1 row)
+
+SELECT pg_column_compression(f1) FROM cmdata3;
+ pg_column_compression
+-----------------------
+ pglz
+ pglz
+ pglz
+(3 rows)
+
-- check data is ok
SELECT length(f1) FROM cmdata;
length
diff --git a/src/test/regress/expected/misc_sanity.out b/src/test/regress/expected/misc_sanity.out
index 9ebe28a78d..4e3b0c13c1 100644
--- a/src/test/regress/expected/misc_sanity.out
+++ b/src/test/regress/expected/misc_sanity.out
@@ -99,6 +99,7 @@ ORDER BY 1, 2;
relname | attname | atttypid
-------------------------+---------------+--------------
pg_attribute | attacl | aclitem[]
+ pg_attribute | attcmoptions | text[]
pg_attribute | attfdwoptions | text[]
pg_attribute | attmissingval | anyarray
pg_attribute | attoptions | text[]
@@ -109,7 +110,7 @@ ORDER BY 1, 2;
pg_index | indpred | pg_node_tree
pg_largeobject | data | bytea
pg_largeobject_metadata | lomacl | aclitem[]
-(11 rows)
+(12 rows)
-- system catalogs without primary keys
--
diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql
index 9d2e72b784..17320c9062 100644
--- a/src/test/regress/sql/compression.sql
+++ b/src/test/regress/sql/compression.sql
@@ -115,6 +115,24 @@ ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4 PRESERVE (pglz2);
SELECT pg_column_compression(f1) FROM cmdata;
\d+ cmdata
+-- compression options
+CREATE TABLE cmdata3(f1 TEXT COMPRESSION pglz WITH (min_input_size '100'));
+CREATE INDEX idx1 ON cmdata3(f1);
+INSERT INTO cmdata3 VALUES(repeat('1234567890',1000));
+SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'cmdata3'::regclass AND attname='f1';
+SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'idx1'::regclass AND attname='f1';
+
+ALTER TABLE cmdata3 ALTER COLUMN f1 SET COMPRESSION lz4 WITH (acceleration '50');
+INSERT INTO cmdata3 VALUES(repeat('1234567890',1004));
+SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'cmdata3'::regclass AND attname='f1';
+SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'idx1'::regclass AND attname='f1';
+
+ALTER TABLE cmdata3 ALTER COLUMN f1 SET COMPRESSION pglz WITH (min_input_size '200') PRESERVE (lz4);
+INSERT INTO cmdata3 VALUES(repeat('1234567890',1008));
+SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'cmdata3'::regclass AND attname='f1';
+SELECT attcmoptions FROM pg_attribute WHERE attrelid = 'idx1'::regclass AND attname='f1';
+SELECT pg_column_compression(f1) FROM cmdata3;
+
-- check data is ok
SELECT length(f1) FROM cmdata;
SELECT length(f1) FROM cmdata1;
--
2.17.0
view thread (430+ messages) latest in thread
Message-ID: <20210218211314.GC14772@telsasoft.com>
Permalink: ../20210218211314.GC14772@telsasoft.com/
Also on: postgresql.org/message-id/20210218211314.GC14772@telsasoft.com
copy link · copy postgr.es · copy git am
reply Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: pryzby@telsasoft.com, robertmhaas@gmail.com, dilipbalaut@gmail.com, tomas.vondra@2ndquadrant.com, a.korotkov@postgrespro.ru, david@pgmasters.net, i.kurbangaliev@gmail.com, 9erthalion6@gmail.com, pgsql-hackers@lists.postgresql.org
Subject: Re: [HACKERS] Custom compression methods
In-Reply-To: <20210218211314.GC14772@telsasoft.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox